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Introduction and Summary of Findings 


National initiatives to raise K-12 standards and align them 
to postsecondary entrance requirements have prompted 
states and/or state postsecondary systems to enact policies 
to improve alignment among postsecondary readiness 
standards, postsecondaiy placement and diagnostic exams, 
and developmental education curricula. 

This report examines how six community colleges— three 
each in the states of Florida and Virginia— responded to the 
state-wide reform of college placement and diagnostic 
testing. It is a companion piece to From Policy to Practice: 
Tracing the Development and Implementation of 
Placement and Diagnostic Assessments across States, 
Systems and Community Colleges - Analysis of Policy 
Reform in Five States that provided an analysis of how five 
states commissioned vendors to develop new placement 
tests that include a diagnostic component. 


Diagnostic assessments are 

designed to provide more specific 
information than traditional placement 
tests about students’ knowledge of a 
particular topic, or capacity and skill 
level in non-cognitive areas. The goal is 
that these assessments will provide 
data on student strengths and 
weaknesses relative to key college 
readiness competencies and based on 
that data, students can access the 
specif c supports they need to 
complete developmental education 
courses more quickly or avoid 
remediation altogether. 


Our analysis in this report provides a 
comprehensive picture of state policy 
reform by tracing implementation through 
multiple levels— beginning most broadly at 
the state level and progressing through 
increasingly granular levels of 
implementation. 

Figure ES-i depicts the levels of 
implementation examined in the report. 


Figure ES-I: Factors Influencing Implementation of Placement and 
Diagnostic Assessment Reforms 


Structure of State and System Policy Levers 


Characteristics of Assessments and Curricular Redesign 


Support and Technical Assistance in 
Implementing Reform 

Institutional Characteristics 
Affecting Implementation 
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Summary of Key Findings 

Table ES-i identifies the most significant elements of the state policy reforms, and identifies the most 
notable institutional and faculty/administrator responses to them. 

Table ES-I: State Policy Decisions and Institutional Responses 


INSTITUTIONAL RESPONSE 

FACULTY/ADMINISTRATOR RESPONSE 

State Policy Decision: Is the assessment reform optiona 

or mandatory? 

• Mandatory adoption led to high levels of integration and scale 
up across the community college system. 

• Faculty and administrators may chafe at the rigidity of the 
requirements, and may create workarounds to retain 
autonomy. 

• Optional adoption led to more limited reform across subsets 
of students and community colleges. 

• Faculty and administrators had the flexibility to customize 
their approaches to adoption which led to less resistance to 
reforms. 

State Policy Decision: How transparent is student placement and diagnostic data? 

• Limited data transparency may hinder the ability of colleges 
to customize student supports and developmental pacing. 

• Faculty preferred previous placement tests when they 
included more actionable student score data. 

• Faculty implemented their own diagnostic tests in class to 
identify individual student needs and verify placement results. 

• The transparency of diagnostic data allows institutions to 
offer curricular support programs with multiple pathways for 
students to complete their developmental education courses. 

• Faculty created developmental education bridge courses and 
reported more effective placement advising when data 
transparency allowed for it. 

State Policy Decision: Has the state developed placement test cut scores below which students cannot enroll in 
developmental education courses? 

• As is the case in Virginia, the creation of a “floor” for 
developmental education eligibility challenged the open- 
access mission of community colleges and has been met with 
resistance across colleges in the state. 

• Faculty resist perceived de-professionalization through the 
reduction of their role in student placement advising. 

• Faculty in two colleges developed “workarounds” to allow 
students below the cut score to receive instruction at the 
college. 

State Policy Decision: To what degree are new tests aligned with specific developmental education 
curricular tools? 

• Strong alignment allows colleges to provide a clear path for 
student progression through developmental education based 
on assessment results. 

• Lack of alignment increases barriers to ensuring a “seamless” 
process of using test results to place students into curricular 
supports that will allow them to complete the developmental 
education sequence quickly. 

• Faculty members prefer diagnostic assessments aligned with 
specific curricular tools, whether they are developed by the 
state or existing vendors, because they allow students to 
access services that address the specific areas where they 
need assistance. 

State Policy Decision: Is the policy change accompanied by adequate implementation support? 

• College administrators in both Virginia and Florida reported 
lack of system-level guidance as a challenge to successful 
implementation. 

• Faculty engagement in peer-to-peer collaboration was 
generally seen as very valuable. 


Lessons from the Field 


Our examination of college-level responses to placement and diagnostic reform in Florida and Virginia 
clearly illustrate that successful implementation of state policy hinges upon a number of conditions and 
factors. Below, we provide a set of recommendations for states to consider as they move towards 
enacting similar reforms. 

Before deciding whether to make testing and diagnostic reform mandatory or optional, 
carefully consider the potential upsides and pitfalls of each. The comparison of Florida’s 
optional adoption policy and Virginia’s mandatory policy provides a clear picture of the pros and cons 
of each approach. State and system policymakers need to weigh their priorities before determining 
which policy direction to embark upon— and develop plans for addressing its challenges. 

Ensure that diagnostic and placement data is as transparent and accessible as possible. It 

makes little sense to enact a large-scale reform in this arena without also ensuring that the data 
generated from the new tests is available to those who are held responsible for serving students— faculty 
and other college staff. 

Align assessment results with existing student data systems. Beyond placement and 
diagnostic purposes, integrating diagnostic data with other college level data systems can be useful for 
evaluation and analyses within developmental education as well as across the college. 

Carefully consider the degree of autonomy that will be granted to institutions and faculty 
in terms of student placement. From a state or system policy perspective, it is more efficient to 
enact across-the-board decisions regarding cut scores and student placement. Yet the efficiencies 
gained must be carefully weighed against the barriers that can be erected when there is inadequate 
flexibility or input from colleges. Rigid placement policies can challenge the open-access mission of the 
community college sector, and could well be met with resistance. 

Invest adequate time and resources to provide the technical assistance and support 
needed for colleges to enact placement and diagnostic testing reforms effectively. As is the 

case when implementing any large-scale reform, accurate information about the reform, and assistance 
for colleges as they implement the policy, can go a long way towards reducing the barriers to reform 
implementation. Creating opportunities for peer-to-peer collaboration could be a particularly effective, 
and relatively low-cost, form of implementation support. 
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Introduction 

Economists have projected the need for an additional 20 million college-educated workers to meet 
future labor market needs. 1 In response, President 
Obama has placed community colleges at the center of 
his education agenda. National foundations have 
followed suit with efforts to raise K-12 standards and 
align these to postsecondary entrance requirements. 

These initiatives have prompted states and/or state 
postsecondary systems to enact policies to improve 
alignment between postsecondary readiness standards, 
diagnostic assessment, postsecondary placement, and 
developmental education curricula. Yet the fidelity of 
implementation of these policies can be variable at the 
institutional level. The purpose of this study is to 
examine how community colleges in two states - Florida 
and Virginia - are responding to new assessment and 
curriculum redesign policies, and to provide a 
preliminary analysis of the perceived effectiveness of the 
assessment and curricular reforms on placement 
accuracy and developmental education completion. 

Research Design and Report Overview 

This report provides findings on the second phase of a two-part research project conducted from 
September 2012 to June 2013: 


Diagnostic assessments are 

designed to provide more specific 
information than traditional 
placement tests about students’ 
knowledge of a particular topic, or 
capacity and skill level in non- 
cognitive areas. The goal is that 
these assessments will provide data 
on student strengths and weaknesses 
relative to key college readiness 
competencies and based on that 
data, students can access the specific 
supports they need to complete 
developmental education courses 
more quickly or avoid remediation 
altogether. 


Phase I (September 20 1 2 to January 20 1 3): During this phase, Research for Action (RFA) 
explored how five states (California, Florida, North Carolina, Texas, and Virginia) reformed placement 
and diagnostic assessments and curricular redesign to increase developmental education completion 


1 Carnevale, A. P., & Rose, S. (2011). The Undereducated American. Washington DC: Georgetown 
University, Center for Education and the Workforce. 



rates. Multiple sources of data were used to develop a picture of these reforms, including a review of 
state requests for proposals (RFPs), document analysis and interviews with state postsecondary leaders. 
Findings from this phase included the following: 

• In most states, the influence of national initiatives and strong state-level leadership, such as 
innovative postsecondary system administrators, came together to spark reform. 

• There was significant overlap across states in terms of assessment design: new tests are to be 
aligned with established academic standards and developmental education curricula; online, 
adaptive delivery is emphasized; and accommodations for special needs students are envisioned. 

• Study states also corresponded in terms of involving faculty in assessment redesign and 
implementation; in requesting ongoing technical support and training from the successful 
assessment vendor; and in the broad strategies associated with complementary developmental 
education reform. Modularization, multiple pathways, and integration of reading and writing 
content into one course are especially prominent. 

• Differences could be found in the length of the tests; timelines for implementation; whether the 
tests were required or offered to institutions for optional administration; and the design 
framework used: college readiness standards-driven assessment design or developmental 
education curriculum-driven assessment design. 

Phase 2 (January to June 20 1 3): Building on Phase l, we examine the implementation of diagnostic 
assessments in two of the states, Florida and Virginia, by studying how state context, policy parameters, 
and institutional factors affect implementation of reforms. However, it is important to recognize that 
this research was conducted during the early phase of reform implementation, and so additional 
research will be needed as implementation progresses to determine whether a comprehensive set of 
policy goals were achieved. This phase has been guided by the following research questions: 

• State and system policy impact at the community college level 

How do external calls for assessment and curricular alignment (e.g., state postsecondary 
readiness standards) affect institutional offerings? 

What is the role of additional training and support— from states, coordinating boards, K-12, 
workforce partners, and other stakeholders— provided to institutions? 

• Institutional perspectives on the implementation and effectiveness of reforms 

How tightly coupled are institutional programs and diagnostic assessments? In what ways 
do institutions incorporate diagnostic assessment data in student-level data systems? 

Do implementation challenges differ when state-developed diagnostic tests are used versus 
those purchased from national vendors? Along what dimensions, and why? 

What are the perspectives of administration, faculty, and students concerning the 
effectiveness of diagnostic assessment systems? What factors do institution stakeholders cite 
as barriers to successful implementation? 

This report is based on a set of cross-site comparative case studies on six community colleges in Florida 
and Virginia (see Appendix A for case studies). Site visits were conducted during the spring 2013 
semester (see Table 1, below). Florida and Virginia were selected as sites because they were the only two 
of the five states from phase 1 that had implemented state-developed diagnostic assessment reforms by 
the spring of 2013. The six community colleges included in this study were selected based on the 
following criteria: 

• Current implementation of the state-developed diagnostic assessment test; 
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• History of developmental education reform with the Achieving the Dream initiative; and 

• Variation in student size and demographics as compared to the other research sites. 

Table I: Community College Study Sites 


Community College Study Site 

Date of Site Visit 

FLORIDA 

St. Petersburg College (SPC) 

March 26-27, 2013 

Tallahassee Community College (TCC) 

March 26-27, 2013 

Valencia College (VC) 

March 20-21, 2013 

VIRGINIA 


Danville Community College (DCC) March 18-19, 2013 

Northern Virginia Community College (NOVA) April 8-9, 2013 

Patrick Henry Community College (PHCC) February 25-26, 2013 


More detailed descriptive information on each institution can be found in Appendix B. 

Interviews and focus groups included faculty, institution administrators, and students, and were 
conducted at the college research sites with protocols developed in consultation with multiple external 
consultants (see Table 2, below). The case studies were reviewed by our primary campus points of 
contact, and each provided feedback to RFA to ensure accuracy and clarity of the community college 
data. 

Table 2: Phase 2 Research Elements 


DATA SOURCES 




FL 

VA 

Institutional Case Studies 

3 

3 

Interviews or Focus Groups with Developmental Education Faculty 

7 

14 

Interviews with Institutional Administrators 

5 

9 

Classroom Observations 

4 

_2 

Focus Groups with Developmental Education Students 

4 

7 


2 Classroom observations were not a part of the research design but were conducted at one Florida college at the suggestion of the college point 
of contact. 
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State and System Policy Context 


This section provides critically important background on state policy and test characteristics relevant 
to our analyses, including the design of the tests, optional or required nature of the assessments, the 
level of data transparency and the competencies or curricula to which the diagnostic tests are aligned 
(see Table 3). 


Table 3: State Assessment Policy 


Vendor 

Selected 

State-Developed Placement & 
Diagnostic Assessment 

Assessment 
Required or 
Optional 

Data & Cut 
Score 

Transparency 

Diagnostic Assessment 
Alignment 

FLORIDA 

McCann 

Associates 

Postsecondary Education 
Readiness Test (P.E.R.T. 
Placement & Diagnostic) 

Optional 

Yes 

Developmental Education 
Competencies 

VIGINIA 

McCann 

Associates 

Virginia Placement Tests 
(V.P.T.) for Math & English 

Required 

No 

Developmental Education 
Curriculum Redesign* 


*The redesign changed the developmental mathematics curriculum so that content was organized into nine units while the 
developmental English curriculum was restructured as an integrated system of reading and writing courses. See discussion 
below for further detail. 


Florida’s Postsecondary Education Readiness Test (P.E.R.T.) 

In order to measure student skills against the state’s Postsecondary Readiness Competencies (PRCs) 
as they enter the Florida College System, the Postsecondary Education Readiness Test (P.E.R.T.) 
system was developed as Florida’s new “preferred” placement test. Florida College System 
institutions began administering the P.E.R.T. in October 2010. The P.E.R.T. Diagnostic is also 
included as another component of the assessment system. While the placement test determines 
appropriate course assignments, the P.E.R.T. Diagnostic identifies, by competency, where skill 
deficiencies exist. The P.E.R.T. Diagnostic has been developed to align with the Developmental 
Education Competencies that are the basis for developmental education courses offered at all Florida 
College System institutions. The goal of the P.E.R.T. Diagnostic is to allow faculty to target the areas 
in which students need additional work. The P.E.R.T. Diagnostic is voluntary for Florida colleges and 
is not computer-adaptive. This means that all students are assessed on the same content because 
questions do not change based on previous responses. Optional developmental education modules 
have also been developed by the Florida Department of Education. 


The P.E.R.T. Placement test includes three computer-adaptive components in reading, writing, and 
mathematics. The P.E.R.T. Diagnostic is a separate component of the assessment system that 
identifies skill deficiencies for students who do not meet the cut scores making them eligible for 
credit-bearing coursework; this element— which is optional for the state’s community colleges— has 
been developed to align with Florida’s Developmental Education Competencies. Unlike the 
placement assessment, the diagnostic tests are not computer-adaptive; every student receives the 
same questions, regardless of previous answers. There are six tests: one for each subject (reading, 
writing, and math) at two competency levels. The reports generated from the P.E.R.T. Diagnostic 
include a student’s score for each test item so that faculty and students can see which skills require 
additional instruction. 
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Virginia Placement Test (V.P.T.) for Math and English 

The Developmental Education Task Force (DETF) was convened in 2008 to review the system’s 
developmental education policies and increase student progress through their courses towards 
graduation; the initiative was supported by involvement in the Developmental Education Initiative. 
The DETF provided recommendations in the report The Turning Point: Developmental Education in 
Virginia’s Community Colleges the following year. In 2010, the Developmental Mathematics 
Redesign Team (DMRT) released recommendations for developmental mathematics courses across 
the Virginia Community College System (VCCS), proposing that the content of the developmental 
mathematics curriculum be revised, with different pathways contingent on a student’s program of 
study; that content be organized into nine pre-college units of study; and that the VCCS develop new 
placement and diagnostic instruments. The Developmental English Redesign Team (DERT) proposed 
that developmental English be restructured as an integrated reading and writing system, with three 
direct pathways to credit-bearing English determined by a student’s placement test score and 
performance against specific learning outcomes. 

The V.P.T. in Math has two, interwoven components: 1) computer-adaptive placement to 
determine whether a student is ready for credit-bearing courses; and 2) non-computer-adaptive for 
students not yet ready for credit-level courses. This second component determines the areas in which 
a student requires additional instruction, and the corresponding developmental education module(s) 
to which they should be directed. When taking the test, students are unaware of when one component 
ends and the next begins. 

The V.P.T. in English likewise consists of two parts: 1) essay; and 2) computer- adaptive placement 
addressing reading and writing. Reading selections are followed by multiple-choice questions to 
evaluate comprehension. Results of the assessment place students in credit-bearing courses, in one of 
three developmental education levels, or require completion of adult basic education before taking 
developmental education courses. The placement results available from the tests do not include 
numeric test scores. The V.P.T. is required across the state. 

Factors Influencing the Implementation of Reforms 

Beyond state context, a number of factors influence the implementation of state-developed diagnostic 
assessments at the community college level. Understanding these factors may help to inform similar 
implementation strategies as the use of diagnostic assessments and aligned curricular reforms spreads 
to other states and systems. Our analysis traces implementation through five levels, beginning most 
broadly at the state level and progressing through increasingly granular levels of implementation; each 
level impacts those below it and can result in responses at the institutional and faculty levels. Taken 
together, they provide a comprehensive picture of how state-level policy rolls out. Figure 1 provides a 
depiction of these levels of implementation. 
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Figure I: Factors Influencing Implementation of Placement and Diagnostic Assessment Reforms 



Structure of State and System Policy Levers 


Characteristics of Assessments and Curricular Redesign 



Support and Technical Assistance in 
Implementing Reform 




Institutional Characteristics 
Affecting Implementation 






Below, we provide our analysis of each of these five levels of implementation. 

I. Structure of state and system policy levers 

High-level policy decisions made during the design of the initiative reverberate through multiple levels 
of implementation. Across our phase 2 study states, three policy decisions related to the creation of 
diagnostic assessments were especially salient: l) whether the reform was mandatory or optional; 2) the 
level of data transparency; and 3) developmental education cut scores. 


Mandatory or Optional Reform: State policy decisions regarding whether to make the use of a 
diagnostic assessment mandatory or optional affects the degree and pace of reform in the state. 
The level of scale-up and the speed with which the diagnostic assessments were implemented were 
determined, in part, by the mandatory or optional nature of the policies. 

VIRGINIA 

In Virginia, mandatory, state-driven reform led to high levels of integration and scale up across all 23 
community colleges in the state with the V.P.T. in math during 2011 and the V.P.T. in English during 
2012. 

FLORIDA 

Florida’s optional, incentive-driven P.E.R.T. Diagnostic strategy has led to slower and more 
incremental reform since it became available in 2012. To date, Florida colleges use the P.E.R.T. 
Diagnostic with a subset of student cohorts and subjects as part of the Florida Developmental 
Initiative Grant from the state department of education. 

Moreover, under legislation passed in the Florida legislature in May 2013 (SB 1720), colleges will no 
longer be able to require students who entered 9 th grade in a Florida public school in 2003-2004 or 
thereafter and who earned a standard Florida high school diploma or active-duty members of the 
United States military to take the state’s placement test or enroll in developmental education courses, 
although the option will be available to them. Older students will not be exempt from placement tests, 
however; if they demonstrate that they need remediation, community colleges will be required to 
offer them a choice between several developmental education options, including “co-requisite” 
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courses in which remedial students get extra help or do additional work in traditional, credit -bearing 
courses. 3 


Whether the reforms were mandatory or optional also determines the degree to which colleges 
can customize adoption. Table 4 provides an overview of how the individual colleges in each state 
responded to the mandatory versus optional approach, using the level of state- and vendor- developed 
assessment implementation across subjects and students as an indicator of the degree of reform 
integration. We define P.E.R.T. and V.P.T. as state- developed tests because they were created as a 
result of state policy. In contrast, we use the term “vendor developed” to refer to tests developed 
solely by vendors such as Pearson and McGraw-Hill, apart from state policy requirements. As can be 
seen below, the mandatory nature of Virginia’s policy resulted in V.P.T. as the sole test across 
campuses, while in Florida the optional testing policy allowed for both state- and vendor-developed 
tests to remain in play. 


Table 4: Level of Assessment Integration in the Study Colleges 



State Developed 
Diagnostic Test 

Math, 

English, Both 

Subset 

All 

Vendor Developed 
Diagnostic Test 

Math, 

English, Both 

Subset 

All 

FLORIDA (optional) 

SPC 

X 

Both 

X 


X 

Both 

X 


TCC 

X 

Math 

X 


X 

Both 


X 

VC 

X 

Math 

X 


- 

- 

- 

- 

VIGINIA (mandatory) 

DCC 

X 

Both 


X 

- 

- 


- 

NOVA 

X 

Both 


X 

- 



- 

PHCC 

X 

Both 


X 






Virginia’s decision to require the implementation of state reforms across community colleges did not 
allow for the degree of customization that was found in Florida, which allowed individual colleges to 
use the P.E.R.T. Diagnostic for a variety of purposes that often supplemented rather than replaced 
existing developmental education initiatives. Specifically: 

• St. Petersburg College: The P.E.R.T. Diagnostic tests are only used for students who want to 
retake the P.E.R.T. Placement exam. Those students take the diagnostic test and must undergo 
tutoring in the area(s) of weakness before they retake the placement exam. 

• Tallahassee Community College: Both the P.E.R.T. and the Assessment and Learning in 
Knowledge Spaces (ALEKS) diagnostic tests are administered at the beginning of a math bridge 
program. 

• Valencia College: The P.E.R.T. Diagnostic in math is offered to students who have scored close 
to the cut-off for credit -bearing coursework on the P.E.R.T. Placement. 


3 Paul Jain, (June 5, 2013). “Remediation if You Want It.” Inside Higher Ed at httD://www.insidehighered.com/news/2Qi3/o6/o.s/florida- 
law-gives-students-and-colleges-flexibilitv-remediation ; The Florida Senate at 
httD://www.flsenate.gov/Committees/BillSummaries/20i3/html/t;oi 
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Data Transparency: Providing diagnostic data as it relates to cut scores is critical to ensure 
accurate student placement, support effective teaching, and provide colleges and faculty the ability to 
individualize support services to students. In Virginia, the V.P.T. was designed to be diagnostic in 
nature, but has provided only placement results. In contrast, both state- and vendor-developed 
diagnostic assessments in Florida allow colleges and faculty to design services for students identified 
through the data. 

Virginia’s lack of transparent diagnostic data led to the following: 

- Faculty reported that the placement- only V.P.T. in English results were a step backwards from 
the previous placement tests when they provided more detailed feedback on student skills. 

- Faculty deem diagnostic data so important that developmental education faculty members in 
math at DCC and English at PHCC implemented their own diagnostic tests to identify individual 
student needs in English and verify V.P.T. placement results in math. 

- Students also reported that they received limited data on their test results after taking the V.P.T. 
One student explained that “I really didn’t get a lot of information.” Another said, “I didn’t even 
know what my score was. They just told me I had to take developmental classes.” Yet a third 
reportedly was told that s/he would “have to take the brush up English course... that was it.” 

Florida’s provision of diagnostic data allows for individualized student supports and placement such as 
the following: 

- St. Petersburg College students scoring at the higher developmental education ranges are offered 
the option to complete an eight-week bridge course or the 16-week developmental education 
course. 

- Valencia College students are provided with the opportunity to take the P.E.R.T. Diagnostic test 
if they score within a particular range on the P.E.R.T. Placement test, but it is optional; they can 
then enroll in one- and two-credit developmental education classes aimed at decreasing the time 
it takes to be eligible for credit-bearing courses. 


Cut scores below which students cannot enroll in Developmental Education: Community 
colleges are mission-driven— faculty and staff strongly believe in open access. The development of 
placement test cut scores that create a “floor” below which students cannot enroll in developmental 
education, and therefore cannot access financial aid, puts that mission at risk. In Virginia, this policy 
decision inspired resistance at the campus level as well as “workarounds” to allow students who 
placed “below the floor” to access developmental education courses. 

Campus Level Resistance: 

Administrators and faculty at all three Virginia study sites expressed concern that the new cut score 
reduced access. One administrator explained: 

I do think they [new cut scores] will have negative consequences for the students... before, we did not 
have a ‘floor’, and the criteria used to determine if we accepted a student would be, ‘does this student 
have the ability to benefit from college courses?’ Well, just about anybody has the ability to benefit. 

So it means we weren’t shutting the door to anybody. 

“Workarounds”: 

As a result of this resistance, colleges have developed ways to allow students who test below the floor 
to enroll in college and receive financial aid. For example, DCC faculty sought funding from the 
community college’s foundation to create short “bridge courses” between semesters to provide extra 
assistance to prospective students scoring at the adult basic level on the V.P.T. This program has 
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already seen some success in preparing students to score well enough on the V.P.T. the second time 
to become eligible for developmental education courses. Similarly, PHCC created a two-credit course 
which teaches students basic computer instruction and whole numbers and a two-credit course that 
helps prepare students for the lowest level of developmental English. These courses allow students to 
access financial aid and avoid adult basic education courses outside of the college. However, since our 
field work in March 2013, VCCS has mandated that students who test below the floor not be allowed 
to enroll in any credit -bearing courses. 


2. Characteristics of the Assessments and Curricular Redesign 

Our analysis also suggested that variations in the specific assessments, and in the scope and nature of 
the curricular redesign that accompanied them, affected the utility of diagnostic assessment systems in 
both states. Below we identify the most salient factors and their impact on implementation. 


Assessment Design: If the test is considered an improvement over what was used before, it is more 
likely to be consistently utilized than if it is not. Test length (as compared to previous placement 
and/or diagnostic assessments) and the integration of placement and diagnostic components 
were two aspects of test design that affected implementation. 

Test Length: 

In both states, community college administrators expressed concern about increased test length, 
especially relative to test fatigue and the college testing infrastructure. For example: 

The P.E.R.T. Diagnostic is reported by faculty and administrators to be longer than off the shelf 
diagnostic tests, which respondents cited as a disincentive to implementation. Further, a 
Valencia College administrator explained that campus testing labs are in high demand; when 
combined with the length of time needed to complete the assessments, capacity and 
infrastructure challenges emerge. Students also expressed concern about the length of the test; 
one stated that, “I thought it would never end,” and another that “it took forever.” 

There was consensus across two of the Virginia college campuses that test length was a 
problem. All faculty respondents, as well as student focus group participants, felt that both the 
math and English tests were too long. Administrators concurred; one interviewee stated: “It’s 
also been challenging because the math test can be really long... upwards of 4 to 5 hours.” An 
administrator at DCC recalled an instance in which the testing coordinator “called me to say, ‘I 
have somebody up here crying taking this test because they’re so stressed, because [the V.P.T.] 
is so long.’ I never had that with COMPASS.” 

Placement and Diagnostic Testing Elements as Integrated or Separate: 

Integrated Assessments: The V.P.T. in math has two, interwoven components administered 
as one test that is designed to include both placement and diagnostic elements. 

Separate Assessments: The P.E.R.T. Placement and P.E.R.T. Diagnostic tests were 
developed and implemented on separate timelines. This factor, coupled with Florida’s optional 
policy, allowed community colleges to select one assessment without the other and administer 
them separately, thereby increasing the number of test administrations overall. 


Degree of Alignment between Diagnostic Assessment and Curricular Redesign: 

Diagnostic assessment data, while useful as a resource to teachers in differentiating instruction, 
becomes more actionable when it feeds into specific curricular strategies to address the skill needs 
identified through the assessment. Therefore, the design of not only the assessment but also aligned 
curricular tools and strategies increases the utility of the reform. For example: 
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FLORIDA 

Existing vendor-developed assessments that continue to be used in Florida’s community colleges 
have been described as more “seamless” than the state-developed P.E.R.T. Diagnostic because they 
link to curricular tools and structures that operationalize the data. Once a student takes the ALEKS 
diagnostic test, for example, s/he knows exactly which skill units need to be completed through the 
online instructional software program aligned with the assessment, and can work independently to 
cover those skills. 

VIRGINIA 

Virginia’s state-developed implementation of the V.P.T. in math has found more success to date than 
the V.P.T. in English because it makes clear connections between skills gaps and specific module 
assignments; the V.P.T. in English provides no diagnostic information, and leads only to one of three 
developmental education courses. 


More Rigorous Curricular Requirements: State requirements in both Florida and Virginia 
regarding the content to be covered and the length of time allotted to cover it have presented 
challenges for faculty and students. 

FLORIDA 

In Florida, the developmental education curriculum has been standardized by the state department of 
education; one faculty member explained that “we have to push every class to get through all the 
material— it is brutal.” 

VIRGINIA 

In Virginia, the modularization of math instruction into four-week units and the integration of both 
reading and writing content in the same course can be difficult for faculty members and students 
alike. Math faculty and campus administrators reported that while successfully completing and 
testing-out of a module is common, and takes place more quickly than with the traditional semester 
courses, it often takes six to eight weeks for students to do so, but is scheduled for only four weeks. 
One faculty member explained: “it might take them six weeks, it might take them eight weeks. On a 
very rare case, it might take them 12 weeks to finish a unit.” An administrator reported that the “math 
faculty say students need more than four weeks to get through some of the units, particularly the 
higher units.” 


Students from one college shared this concern about the press for time to complete developmental 
coursework under the math curriculum redesign. 


Four weeks for one module - it is way too quick. Some of these modules are four or five 
chapters long. 

The teacher is so excited that you only have 4 weeks of this and you gotta get it. Fm like who’s 
going to help me get it because I can’t get math this way. I need someone to come help me. 

Fm on module one and it’s going from like... fractions to dividing them ... Fm like ease up. I need 
some time. 
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Similarly in English, the combination of reading and writing content in the same course has made 
covering the content a challenge over the semester. One English faculty member explained: “we are 
feeling lots of pressure. This has been difficult to do; we have folks who have always been full-time 
reading faculty, others who had writing specialties, so we’ve had to help people combine those two 
things.” In describing the integration of reading and writing in the same course, another English 
faculty member said: “a challenge is an understatement.. .forcing that much information onto a 
developmental education student.” 


3. Support and Technical Assistance in Implementing Policy 

Implementing large-scale reforms in developmental education requires initial training and ongoing 
technical assistance for both administrators and faculty. State-level research during phase l of the 
project identified “communicating with college staff’ and “including faculty in the process” as 
challenges for reform; our institutional case work and campus interviews confirmed this. Informal 
collaboration among peers emerged as a promising practice. 


In both Florida and Virginia, faculty and administrators from two of the three study 
colleges reported limited direct support from the state department or system office: 

FLORIDA 

Due in part to the voluntary nature of the P.E.R.T. Placement and Diagnostic assessments, both 
faculty and administrators described the information provided by the state department as limited; 
the Florida Developmental Initiative Grant was a notable exception. 

VIRGINIA 

One administrator described support from the system office as rote guidance on “a new hard-and-fast 
rule,” as opposed to meaningful training in support of the initiative. Another claimed the college saw 
“nothing from the state as far as the diagnostic test is concerned.” The system office communicated 
with colleges through liaisons but provided them few resources to share with local campuses, such as 
sample materials or presentations. 


Collaboration with peers was identified as a valuable form of training on the curricular 
implementation. Faculty and administrators at both DCC and NOVA discussed the importance of 
meeting with peers from across the state to discuss barriers and successes in implementing the new 
curricular redesign. One administrator at Danville explained: “the highlight of the meeting for us was 
interacting with everyone else to see ‘how are you managing to teach three units in one classroom?’ 
Those takeaways are good for us.” 




4. Institutional Characteristics Affecting Implementation 

State or system-level developmental education reforms are enacted within the context of existing 
college-level practices and conditions. These factors affect the roll-out of diagnostic assessments. Key 
institutional-level facilitators and barriers include: l) relationships with existing vendors, 2) the history 
of advising and placing students at the college level, and a 3) lack of connection across student data 
systems. 


Relationships with Existing Vendors: The three Florida colleges included in this study already 
had experience working with, and receiving training from, testing and curriculum vendors such as 
McGraw-Hill and Pearson at the time of the P.E.R.T. Diagnostic roll-out. These relationships 
influenced their decisions regarding the level of adoption and implementation. For example: 

McGraw-Hill has worked with Tallahassee Community College for at least five years and 
consulted with the college to adapt the ALEKS to meet state standards and curricular 
requirements before the P.E.R.T. Diagnostic was implemented. This history with McGraw-Hill 
and their willingness to provide technical assistance to address state curricular requirements 
encouraged college faculty to continue to use the ALEKS diagnostic test and online tools. 
Pearson trained faculty at Valencia in the use of the Foundations of Math software, and how to 
integrate the P.E.R.T Diagnostic results into module assignments. 


College History of Advising and Placing Students: College faculty and administrators in 
Virginia have traditionally made placement decisions at the campus level, informed but not dictated 
by placement score data. The implementation of the V.P.T. essentially short-circuited this function, 
since test scores automatically placed students into specific courses. This was true despite the lack of 
diagnostic data, which has caused concern on college campuses: 

At PHCC, one administrator explained that her inability to understand how close a student tests 
to the cut score on the V.P.T. prevents her from advising students appropriately. Another 
explained that the COMPASS had provided scores that allowed advisors to see how close the 
student was from the cut score, and then work with that student on what resources s/he needed 
to progress onto college level coursework. 

At NOVA, placement test results had been used to advise students, but the data provided by the 
V.P.T., especially in English, do not lend themselves to this process. One English faculty 
member explained that the college “had a good system in place [before the V.P.T.] in looking at 
multiple measures... to move students into credit-bearing classes when they were ready.” 


Lack of Connection across Student Data Systems: Institutions in both states did not link 
diagnostic assessment results to other student data records at the college level, limiting the potential 
for data to inform and refine practice beyond the immediate developmental education program (see 
Table 5, below). 
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Table 5: Diagnostic Data Collection and Use 


Community College 

Data identifies 
individual student 
learning needs 

Diagnostic test 
results included in 
student data system 

Test data linked to 
other student records 

Data used for 
additional purposes 

FLORIDA 

SPC 

O 

® 

® 

® 

TCC 

O 

® 

® 

® 

VC 

O 

® 

® 

® 

VIRGINIA 

DCC 

Math Only 

O 

® 

® 

NOVA 

Math Only 

O 

® 

® 

PHCC 

Math Only 

O 

® 

® 


Limited potential for analysis and evaluation using diagnostic data: 

Without the ability to link student data systems that house diagnostic assessment data with other 
student records, or across multiple student cohorts, college administrators are unable to look at 
trends in student performance. None of the colleges in either state linked diagnostic data to other 
data records. However, it should be acknowledged that the primary purpose of placement tests has 
traditionally been to accurately assign students to the correct level of coursework— not to conduct 
longitudinal data analysis or evaluation. 

Tracking student data manually: 

Interest in such data analysis has led to manual tracking of student progress in multiple Florida sites, 
despite the difficulties involved. Specifically: 

- At SPC, administrators have begun to track students after bridge course completion and 
tabulate the college success rates. 

At TCC, developmental education faculty members compile individual student data to 
determine student progress, though the process is reportedly cumbersome. 


5. Perceptions of the Effectiveness of Reforms 

Buy-in among faculty and administrators is central to the success of reform. Student perceptions and 
experiences are also an important indicator of a reform’s effectiveness, and can point the way to 
important improvements. Two indicators of faculty and administrator support were identified as 
particularly salient: faculty perceptions of the effectiveness of: l) the new assessments; and 2 ) the 
aligned curricular reforms to improve student completion. 


Perception of Placement Accuracy: While not a primary focus of state assessment reforms, poor 
placement has been an ongoing concern with traditional placement testing due to emerging research 
that questions their accuracy. Faculty perceptions as to whether the new tests improve student 
placement results are critical to implementation success. Table 6, below, summarizes how faculty 
members and administrators at each of the six community colleges assess whether the new tests 
improve placement accuracy. 


13 





Table 6: Do faculty and administrators believe P.E.R.T /V.P.T. improved placement accuracy vs. off-the-shelf tests? 


FLORIDA STUDY SITES (P.E.R.T) 

VIRGINIA STUDY SITES (V.P.T.) 

SPC 

TCC 

VC 

DCC 

NOVA 

PHCC 

Math 

Eng. 

Math 

Eng. 

Math 

Eng. 

Math 

Eng. 

Math 

Eng. 

Math 

Eng. 

® 

® 

® 

® 

TBD 

TBD 

o 

Mixed 

® 

® 

o 

Mixed 


FLORIDA 

Generally speaking, no faculty and administrators at the study sites reported that the P.E.R.T. 
improved placement accuracy. They were either unsure of whether the P.E.R.T. Placement test 
increased placement accuracy, or felt that the P.E.R.T. had not improved the accuracy of placement 
results. Placement in support programs, such as bridge courses that target students close to the 
eligibility for credit -bearing work, was seen as especially problematic: 

At SPC, students nearest to the cut-point for credit-bearing coursework were enrolled in the 
first semester of the My Bridge to Success program, which offers tailored instruction through 
the use of online tools in developmental reading, writing, and math to help students reach 
transfer-level courses. However, since the first semester of the program’s implementation 
students who place in the upper developmental writing and reading levels have been permitted 
to self-select the program, regardless of their P.E.R.T. Placement scores, due to the lack of 
correlation between P.E.R.T. Placement scores and student success in the program. 

At TCC, the rigor of the P.E.R.T. Placement test has been compared with that of the ALEKS test 
based on the college’s experience with students in the math bridge program. While the number 
and exact content of the topic areas vary between the ALEKS and P.E.R.T., students placed in 
the program have scored close to the cut score for credit-bearing coursework on the P.E.R.T. 
Placement test, but have commonly been found to need considerably more remediation across a 
larger number of topics based on the ALEKS test. 


VIRGINIA 

Perceptions of the accuracy of the V.P.T. were mixed across colleges and subjects. The placement and 
diagnostic components are included in one test, making the accuracy of the V.P.T. central to 
successful implementation. While respondents at the two more rural southern colleges were generally 
positive in assessments of test accuracy, faculty at NOVA were more negative. Specifically: 

Math faculty at both DCC and PHCC said that the development of nine math modules 
improved student placement. Faculty at NOVA, however, argued that students were often 
placed in college-level courses for which they were not prepared. Students enrolled in 
developmental math modules in Virginia expressed similar concerns. Students at DCC and 
PHCC reported that they felt they had been placed appropriately, while at NOVA a student 
explained that “I thought [the V.P.T.] over placed me at first because I had signed up for pre- 
calculus in high school but I was only in it for two days. ..so for [the V.P.T.] to come back and 
say I was placed in pre-calculus just didn’t seem right.” 
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English faculty members were less favorable in their assessment, and expressed widespread 
skepticism about test accuracy. Multiple respondents felt that students were being placed above 
their ability levels. Students had mixed reviews— some felt that they were placed appropriately 
and others did not. 


Perception of the Effectiveness of Accompanying Curricular Reforms: The purpose of the 
diagnostic assessments is to provide data to target and improve learning and instruction, which has 
been operationalized through curricular redesign. Therefore, perceptions of the effectiveness of the 
curricular redesign are connected to the success of test implementation. 

MATH: 

In both states and across all but one college, there was consensus that math modularization has been 
effective in allowing students to move through developmental math more quickly. For example: 

FLORIDA 

Study colleges continued to implement vendor-developed (i.e., McGraw-Hill, Pearson) online 
curricular tools in math in which students found success. At SPC, students reported their approval of 
the modularized approach, and expressed a desire to continue using curricular software moving 
forward into college-level coursework. There was consensus at TCC that the ALEKS program has 
been effective in moving students through developmental education courses in math more quickly, 
and both faculty and administrators reported that the data clearly show that students who complete 
developmental education with ALEKS show stronger fundamental math skills than students who do 
not. At Valencia, students reported that the software motivated them to stay on top of their work, and 
that they preferred the modular approach over the traditional, in-person, classroom-based approach. 
These responses provide evidence that existing curricular tools may already address the outcomes to 
which states and systems aspire, and that the creation of additional state-developed curricular tools 
may not be necessary in all cases. 

VIRGINIA 

Redesign of the developmental math curriculum created nine modules that focus on discrete, key 
skills. The module(s) a student takes depends on their academic goals and V.P.T. results. For 
instance, liberal arts students are only required to prove mastery in modules l through 5, while 
science and math students are required to complete all 9 modules successfully. There was consensus 
among faculty at DCC that the process of completing the developmental education requirements is 
“much faster than it was” and in some cases “students just blaze through these [math] modules.” 
NOVA respondents recognized that redesign established different exit points for students and 
allowed for more customized course-taking. At PHCC, however, because of the high passing rates 
before the redesign, the college has experienced a decrease in the percentage of students passing 
developmental education modules. 

Student perceptions about the effectiveness of the math modules were mixed as well. With the 
appropriate level of self-motivation, they worked well: 

I like it, because even though some people might be behind, I can still move forward. You just 
have to have the motivation to do it. That’s probably the main problem. I’m ahead of where I’m 
supposed to be, so I like it. 
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The developmental class is very helpful. It is very effective - if you are dedicated enough you 
can just power through the units one at a time and don’t have to spend a lot of time reviewing. 

Yet those who reported a lack of discipline or the need for additional help found the modular format 
more problematic: 


I don’t like it because I procrastinate a lot , and that gives me a chance to procrastinate, because 
you have to do it yourself. And she’ll tell you that you need to work on this, but I don’t do it. 
Outside of class, I don’t work on it. 

The last module I’m kind of stuck on right now. I’m getting ready to take my last one over. It’s 
kind of frustrating . I don’t like the self-paced without enough help. I need to get right there face 
to face. 


ENGLISH: 

Available data on the implementation of developmental education redesign was more limited due to 
the focus on implementation of the P.E.R.T. Diagnostic in math, and the more recent implementation 
of the V.P.T. in English. 

FLORIDA 

Only one college provided information on developmental education redesign in English. At SPC, 
students who self-select into the modular developmental courses complete a diagnostic exam offered 
through the course’s computer software (i.e., Pearson’s MyWritingLab and MyReadingLab). Faculty, 
as well as students, shared enthusiasm for the modularized approach, but cautioned that students 
must be motivated and have the ability to work independently. Administrators provided data 
demonstrating higher pass and lower withdraw rates compared to the traditional developmental 
courses. 

VIRGINIA 

The redesign integrated reading and writing courses into three placement levels. With 
implementation starting just this year, it is too early to report on whether students’ progress more 
quickly than under previous curricula. 


Lessons from the Field 

Our examination of college-level responses to placement and diagnostic reform in Florida and Virginia 
clearly illustrate that successful implementation of state policy hinges upon a number of conditions and 
factors. Below, we provide a set of recommendations for states to consider as they move towards 
enacting similar reforms. 

• Before deciding whether to make testing and diagnostic reform mandatory or 

optional, carefully consider the potential upsides and pitfalls of each. The comparison 
of Florida’s optional adoption policy and Virginia’s mandatory policy provides a clear picture of the 
pros and cons of each approach. State and system policymakers need to weigh their priorities 
before determining which policy direction to embark upon— and develop plans for addressing its 
challenges. 
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• Align assessment results with existing student data systems. Beyond placement and 
diagnostic purposes, integrating diagnostic data with other college level data systems can be useful 
for evaluation and analyses within developmental education as well as across the college. 

• Ensure that diagnostic and placement data is as transparent and accessible as 
possible. It makes little sense to enact a large-scale reform in this arena without also ensuring that 
the data generated from the new tests is available to those who are held responsible for serving 
students— faculty and other college staff. 

• Carefully consider the degree of autonomy that will be granted to institutions and 
faculty in terms of student placement. From a state or system policy perspective, it is more 
efficient to enact across-the-board decisions regarding cut scores and student placement. Yet the 
efficiencies gained must be carefully weighed against the barriers that can be erected when there is 
inadequate flexibility or input from colleges. Rigid placement policies can challenge the open- 
access mission of the community college sector, and could well be met with resistance. 

• Invest adequate time and resources to provide the technical assistance and support 
needed for colleges to enact placement and diagnostic testing reforms effectively. As 

is the case when implementing any large-scale reform, accurate information about the reform, and 
assistance for colleges as they implement the policy, can go a long way towards reducing the 
barriers to reform implementation. Creating opportunities for peer-to-peer collaboration could be a 
particularly effective, and relatively low-cost, form of implementation support. 


Next Steps for Research 

These early findings provide important feedback to a range of stakeholders and interested parties as 
they implement, or contemplate implementing, broad-scale assessment and placement reforms. The 
findings also strongly suggest that tracking policy implementation requires examining the process 
through multiple lenses and units of analysis, including state, institution, staff and student perspectives. 
Yet it is too early to know whether initial challenges identified will be addressed moving forward; and 
how variation in state approaches and contexts will play out over time. 

As reforms move towards full implementation, it will be critical to continue tracking the following 
factors, which this initial research suggests play a critical part in the rollout of developmental education 
and assessment and placement reforms. The following addresses each of the factors identified and pose 
continuing questions for the research moving forward: 

Structure of State and System Policy Levers 

• How does the pace and degree of reform change over time in states where these initiatives are 
mandatory versus optional? 

• How do community colleges address issues of data transparency in order to access the information 
they need to provide specific instructional supports for students? 

• Will states continue to create cut scores below which students cannot enroll in developmental 
education and expand options for students to avoid developmental education; if so, how do colleges 
address access? 

• What is the relationship between K12 Common Core State Standards and assessments introduced 
2012-2014 and college-level assessment and placement policies? 
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Characteristics of Assessment and Curricular Redesign 

• How do new, state-developed diagnostic assessments compare in design to off-the-shelf tests, and 
how has design impacted the adoption of reforms and implementation decisions? 

• How are states, vendors, and community colleges addressing the need for alignment among K12 
Common Core State Standards and related assessments, college-level diagnostic assessments, 
developmental education curricular tools, and college-level curricula? How have curricular 
requirements at the state and system levels changed, and how do they impact implementation of 
curricular redesign? 

Support and Technical Assistance in Implementing Reform 

• What supports are states providing community colleges to assist in implementing new assessment 
systems and curricular reforms? Do colleges find them to be effective? 

• What types of supports would community college administrators and faculty find most helpful in 
supporting the work? 

• What information and supports will increase awareness and help students prepare for and 
successfully navigate the changing requirements for college assessment and placement? 

Institutional Characteristics Affecting Implementation 

• How can, and do, states allow colleges to include existing practices, assessments and curricular tools 
to address emerging policy requirements? 

• How do existing college practices support and hinder the adoption of reforms? How do both college 
structures and state policies change over time as a result? 

• How can assessment data become integrated with other data systems to inform both developmental 
education and wider college practice? 

Perceptions of the Effectiveness of Reforms 

• How have placement results from new assessment systems impacted the experiences of students in 
both developmental education and credit -bearing courses? 

• How have faculty members used diagnostic information to differentiate instruction? How have 
community colleges used it to provide curricular resources and support services to address student 
needs? 

• What have been the results of curricular reforms on the ability of students to complete the 
developmental education sequence and be successful in credit-bearing classes? 

Answers to these and other questions can provide a more definitive measure on the success of large- 

scale assessment and curricular reforms at the system, institution, and student levels. 
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Appendix A. Case Studies 



RESEARCH for ACTION 


Developmental Education and Diagnostic Assessment Reform 
Community College Case Study: St. Petersburg College (SPC) 


The following profile has been developed to outline the 
local context and implementation of developmental 
education and diagnostic assessment reform at St. 
Petersburg College. The profile was verified by the point of 
contact at the college; data included in the profile offers a 
stand-alone summary and will be used for cross-site 
analyses. The profile addresses six areas: l) local and 
community college context, 2) overview of the research, 

3) integration of placement and diagnostic assessment, 

4) placement and diagnostic assessment data use, 

5) perceptions of the effectiveness of the assessment system 
and developmental education program, and 6) challenges 
to implementation. The following research questions 
guided the site visits at the community college: 

• How do external calls for assessment and curricular 
alignment (e.gf., state postsecondary readiness 
standards) affect institutional offerings? 

• What is the role of additional training and support— 
from states, coordinating boards, K-12, workforce 
partners, and other stakeholders— provided to 
institutions? 

• How tightly coupled are institutional programs and 
diagnostic assessments? In what ways do institutions 
incorporate diagnostic assessment data in student-level 
data systems? 

• Do implementation challenges differ when state- 
developed diagnostic tests are used versus those 
purchased from national vendors? Along what 
dimensions, and why? 

• What are the perspectives of administration, faculty, 
and students concerning the effectiveness of diagnostic 
assessment systems? What factors do institution 
stakeholders cite as barriers to successful 
implementation? 


Florida State Context 

In order to measure student skills against 
the state’s Postsecondary Readiness 
Competencies (PRCs) as they enter the 
Florida College System, the Postsecondary 
Education Readiness Test (P.E.R.T.) system 
was developed as Florida’s new 
“preferred” placement test. Florida 
College System institutions began 
administering the P.E.R.T. in October 
2010. In addition to P.E.R.T. Placement, the 
P.E.R.T. Diagnostic is another component 
of the assessment system. While the 
placement test determines appropriate 
course assignments, the P.E.R.T. Diagnostic 
identifies, by competency, where skill 
deficiencies exist. The P.E.R.T. Diagnostic 
has been developed to align with the 
Developmental Education Competencies 
that are the basis for developmental 
education courses offered at all Florida 
College System institutions. The goal of 
the P.E.R.T. Diagnostic is to allow faculty 
to target the areas in which students need 
additional work. The P.E.R.T. Diagnostic is 
voluntary for Florida colleges and is not 
computer-adaptive, so that all students are 
assessed on the same content as questions 
do not change based on previous 
responses. Optional developmental 
education course modules have also been 
developed by the Florida Department of 
Education. 
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I. Local and Community College Context 
Local Context 

St. Petersburg College is based in St. Petersburg, in Pinellas County, Florida. St. Petersburg and 
Clearwater are adjacent to Tampa, home of the University of South Florida, and are part of the Tampa 
Bay Area, the second-largest metropolitan statistical area in Florida. The health care, retail, tourism, 
and manufacturing sectors are the largest employers in the area. Table l shows that the 2012 estimated 
population of Pinellas County is over 920,000, primarily white (76.5%), with African Americans 
(10.7%) and Hispanic/Latino(a)s (8.3%) as the largest minority populations. The majority of residents 
have high school diplomas, but fewer than 30% have four-year degrees. 

Table I: Population and Demographics 


POPULATION 

Total Population 

921,319 

Percentage of Population with High School Diploma 

88.3% 

Percentage of Population with Four-Year Degree 

27.0% 

Average Household Income 

$45,891 

DEMOGRAPHICS 

Percentage White 

76.5% 

Percentage African American 

10.7% 

Percentage Hispanic 

8.3% 

Percentage Asian 

3.1% 

Percentage Other 

2.3% 


Source: U.S Census Bureau State & County Quick Facts at http://quickfacts.census.gov/qfd/states/ 12/121 03.html 


Community College Context 

St. Petersburg College was opened in 1927 and currently has 11 campuses/sites in Pinellas County. The 
college serves the postsecondary education needs of students from Pinellas County and the greater 
Tampa Bay Area. In 2002, SPC began offering baccalaureate degrees and was the first of Florida’s 
community colleges to expand into the four-year sector, enrolling over 5,000 students in 22 programs 
by 2010. 

The majority of students enroll in Associate of Arts (53%) or Associate in Science (24%) degree 
programs, with just over 4% seeking a certificate; 76% of St. Petersburg College students who transfer 
to state universities in Florida have earned an Associate of Arts degree. In total, SPC is a large, multi- 
campus college serving over 50,000 students annually with over 370 instructional staff. SPC is an 
Achieving the Dream Participating Institution and has focused on improving student outcomes and 
increasing access for low-income, first-generation, disabled, and underrepresented students. In its 
efforts to improve outcomes and course completion rates, SPC sought and secured funding from the 
Florida Developmental Initiative Grant in spring 2010 to develop the My Bridge to Success program. 
My Bridge to Success offers individually tailored instruction through the use of media-based tools in 
developmental reading, writing and math to help students reach transfer-level courses. While the My 
Bridge courses are offered on five SPC campuses, this study was conducted on the Clearwater Campus 
with faculty and staff from multiple campus locations participating. 
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Table 2: Student Characteristics and Academic Programs, 201 1-12 


STUDENT CHARACTERISTICS 

Unduplicated Headcount 

58,759 

Total Full-Time Equivalent (FTE) Enrollment 

22,407 

Percentage White Students 

68% 

Percentage African-American Students 

14% 

Percentage Hispanic Students 

8% 

Percentage Other Students 

10% 

Full-Time Beginning Undergraduates Receiving Financial Aid 

86% 

ACADEMIC PROGRAMS* 

Number of Bachelor’s Degree Programs (and Graduates) 

22 (1,033) 

Number of Associate Degree Programs (and Graduates) 

42 (3,624) 

Number of Certificate Programs (and Graduates) 

66 (5 1 8) 

Number of Diploma Programs (and Graduates) 

2(96) 


*Data for academic programs and graduates are from 20 1 0-20 1 I 

Sources: St. Petersburg Community College Fact Book ( http://www.spcollege.edu/central/ir/Web Factbook I I - 1 2.pdf ) and Florida 
Department of Education Community College Fact Book ( http://www.fldoehub.Org/CCTCMIS/c/Documents/Fact%20Books/fb20 1 2.pdf ) 


II. Overview of the Research 

Research for Action (RFA) contracted with three representatives from Florida State University to 
conduct a field work site visit (see Table 3) in the spring semester of 2013 to explore the 
implementation and impact of the My Bridge program and the use of the P.E.R.T. Placement and 
Diagnostic assessments on math, reading, and writing developmental education instruction. 


Table 3: Research Conducted 

DATA SOURCES 

Interviews with Administrators I 

Developmental Education Mixed-Faculty Focus Groups 2 

Math Developmental Education Student Focus Groups 2 

Writing Developmental Education Student Focus Groups I 

Math Classes Observed 2 

Writing Classes Observed I 

Reading Classes Observed I 


In addition, we conducted background research using institutional websites, documents, and phone and 
email conversations with college administrators. 


21 





III. Integration of Placement and Diagnostic Assessment at the Community College 

Both vendor-developed (“off the shelf’) and state-developed assessment instruments have been used to 
varying degrees across the college, and are discussed below (see Table 4): 


Key for Table 4 through Table 6 

© YES 

( 

8 

) NO 


Table 4: Level of Diagnostic Assessment Integration within the Community College 


STATE-DEVELOPED DIAGNOSTIC ASSESSMENT (P.E.R.T. DIAGNOSTIC) 

Administered in Math? 


Administered in English? 


Fully Implemented with All Students? 


VENDOR-DEVELOPED DIAGNOSTIC ASSESSMENT 

Administered at the Community College? 

o 

Administered in Math? 

© 

Administered in English? 

o 

Fully Implemented with All Students? 

(N). 


*Students who desire to retake the P.E.R.T. Placement test are required to first take the P.E.R.T. Diagnostic. These are the only students 
to whom the P.E.R.T. Diagnostic is administered. 

**While only students near the cut-point for developmental education were enrolled in the first semester of My Bridge courses (spring 
201 I), students who place in the upper developmental writing and reading levels have been permitted to self-select (regardless of P.E.R.T. 
scores) since fall 2011. 


Integration across Subjects and Students: 

P.E.R.T.: 

Placement: The placement tests are implemented college-wide. SPC requires all entering 
students to take the P.E.R.T. Placement tests in math, reading, and writing unless the 
students provide: qualifying SAT or ACT scores; other placement scores not older than two 
years; or evidence of college-level course credit in the respective subject area with a grade of 
“D” or better. Students scoring in the higher-level developmental education ranges are 
offered the option to complete the 8-week bridge course or the 16-week developmental 
courses. 

Diagnostic: The P.E.R.T. Diagnostic is only used for students who want to retake the P.E.R.T. 
Placement exam. Those students take the Diagnostic and must undergo tutoring (generally 
administered through the learning center) in the areas of weakness before they retake the 
placement exam. SPC does not use the P.E.R.T. Diagnostic within the modular classes 
because they developed My Bridge prior to the release of the P.E.R.T. Diagnostic exams. SPC 
administrators also expressed concern that the P.E.R.T. Diagnostic exams are linear, paper 
and pencil exams, as opposed to adaptive computer-based exams. 
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- Vendor-Developed Assessments and Software Tools: Students who self-select into the modular 
developmental courses complete a diagnostic exam offered through the course’s computer 
software (Pearson’s MyWritingLab and MyReadingLab and McGraw-Hill’s ALEKS). The vendor 
diagnostics are completed on the first day of the course and allow the software to tailor class 
modules to the students’ specific areas of need. 

Impact on College Policy and Practice: The use of the P.E.R.T. Placement and Diagnostic tests has 
fundamentally altered developmental education at SPC. Administrators indicated that after receiving 
the Florida Developmental Initiative Grant, they made the choice to implement diagnostic assessments 
and the modularized curricula in all three developmental areas (reading, writing, and math). Since the 
initial implementation, SPC has continued to draw down the number of 16-week courses to drive 
student enrollment into the 8-week modularized courses. 

Impact on Instruction: The use of the diagnostic exams and the modularized course approach has 
influenced change in classroom instruction and pedagogy. The modularized approach has led to faculty 
reducing or even eliminating lectures, and instead using class time to monitor student progress through 
modules and offer individual and/or group assistance to students. 

IV. Assessment Data Use 

The use of diagnostic test data from both the P.E.R.T Diagnostic and other, vendor-developed 
diagnostic tests is outlined in Table 5 below and explores the extent to which the data informs 
developmental education as well as other college programs. 


Table 5: Use of Diagnostic Assessment Data 


STATE-DEVELOPED DIAGNOSTIC ASSESSMENT (P.E.R.T. DIAGNOSTIC) 

MATH 

ENGLISH 

Are the diagnostic test results included in the college student data system? 

® 

® 

Is the test providing data that can be used to identify individual student 
learning needs? 

O 

O 

Are the test data linked to other student records? 

® 

® 

Are the data being used outside of student placement and diagnostic 
decisions in developmental education? 

® 

® 

VENDOR-DEVELOPED DIAGNOSTIC ASSESSMENT 



Are the diagnostic test results included in the college student data system? 

® 

® 

Is the test providing data that can be used to identify individual student 
learning needs? 

O 

O 

Are the test data linked to other student records? 

® 

® 

Are the data being used outside of student placement and diagnostic 
decisions in developmental education? 

® 

® 


Inclusion in the Student Data System: Diagnostic data is not incorporated into the college data 
system. Despite the widespread use of diagnostic assessment data in SPC’s developmental education 
programs, it is not included in the institution’s data warehouse. 
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Identification of Individual Student Learning Needs: Both the state- and vendor-developed 
assessments provide data on individualized student learning needs, disaggregated by subtopics 
included on the assessment. 

Linkage to Other Student Data: Diagnostic data is not incorporated into data reports on other 
student characteristics at this time. SPC has, however, has undertaken an ongoing project to manually 
track students completing the bridge courses. 

Data Use outside of Developmental Education: Diagnostic data is not used outside of the 
developmental education program. SPC’s data warehouse does not store diagnostic data, so any 
reporting of diagnostic data and student records is done manually by campus administrators. The 
course software tools do offer reporting on individual students within courses, but connecting these 
data to existing institutional records must be done by hand. SPC administrators have begun a project to 
track My Bridge students after course completion and tabulate their college success rates. 

V. Effectiveness of the Assessment System and Developmental Education Program 

The reported level of effectiveness varied among the different placement and diagnostic assessments 
used at SPC: 


Table 6: Respondent Perceptions of the Effectiveness of Assessments and Curriculum Redesign 


STATE-DEVELOPED ASSESSMENT (P.E.R.T.) 

MATH 

ENGLISH 

Is the test placing students more accurately in the developmental education 
sequence? 

® 

® 

Is the curriculum redesign allowing students to complete the developmental 
education sequence more quickly? 

N/A 

N/A 

VENDOR-DEVELOPED DIAGNOSTIC ASSESSMENT 



Is the test placing students more accurately in the developmental education 
sequence? 

O 

O 

Is the curriculum redesign allowing students to complete the developmental 
education sequence more quickly? 

O 

O 


Student Placement The P.E.R.T. Placement test alone has had little impact on developmental reading, 
writing, or math placement. P.E.R.T., like Accuplacer before it, simply placed students into 
developmental education; the combination of diagnostic assessments and a modularized approach 
through software, which is provided only by the vendor-developed assessments, seems to have had the 
greatest impact on student success. Additionally, administrators and faculty were not entirely pleased 
with the P.E.R.T. Placement test. Administrators shared that they found no correlation between 
P.E.R.T. Placement scores and student success in the bridge courses, leading them to open the bridge 
courses to all students in the higher development levels as opposed to only the top scorers. Faculty also 
expressed concern that they were seeing more unprepared students in college-level courses as a result 
of P.E.R.T., as compared to Accuplacer, with one faculty member commenting that placement accuracy 
is “not even close to what it used to be.” 

Diagnostic Assessment SPC implemented “off the shelf’ diagnostic exams on a large scale and saw 
largely positive results; the P.E.R.T. Diagnostics are used on a limited basis. Students generally felt the 
diagnostics from Pearson and McGraw-Hill provided an accurate assessment of their skills and 
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deficiencies. Faculty tended to agree, indicating that the diagnostics offer “validation” for the students 
in areas they excel and a “realization that they need to pay attention” in areas where they are weak. 

Developmental Course Completion: Students complete the semi-self-paced developmental 
education program through the use of the course’s computer software and under the tutelage of a 
faculty member. Students completing the bridge courses are permitted to exit early and enroll in 
college-level coursework in the same term. Students are required to attend a twice-per-week, in-person 
class. Students from all three subject areas shared their unanimous approval of the modularized 
approach, and expressed their desires to continue using software moving forward into college-level 
coursework. One student remarked that using the software and completing modules was “as addictive 
as gambling.” Faculty shared similar enthusiasm, although they cautioned that students must be 
motivated and have the ability to work on their own to be successful. Administrators also indicated their 
support, and provided data demonstrating higher pass rates and lower withdrawal rates for the bridge 
courses compared to the traditional developmental courses. 

Supports to Increase Effectiveness 

- Both McGraw-FIill and Pearson have worked with SPC in designing the course competencies 
and in training faculty to use the software. Faculty in particular expressed great appreciation of 
the support from vendors and their campus representatives in addressing faculty needs, 
including having flexibility with the use of software access codes for students using financial aid. 

College administrators have supported the initiative. SPC’s former president had expressed 
concern about pass rates for students, leading other administrators to seek methods for 
improvement, and SPC’s current president has led the charge for SPC to become an Achieving 
the Dream Participating Institution. The administration also matched the $30,000 funding 
from the initiative. 

VI. Challenges to Implementation 

A number of issues were raised as barriers to implementation of the P.E.R.T. Placement and Diagnostic 
systems: 

The P.E.R.T. Placement test is not placing students as accurately as Accuplacer. 

Although students generally felt they had been adequately placed, faculty expressed concern that the 
P.E.R.T. Placement tests were placing students who would have previously been placed in 
developmental education into college-level courses. 

Students do not always understand the options available to them, and many are intimidated by 

the modularized approach Student focus group participants recalled seeing classmates leave on the 
first day of class when they learned of the class format with the self-paced modules. Faculty also 
identified that communication and advertising to students is a challenge, and that the length of the 
diagnostic exams may intimidate some students. 

The reliance on computers for instruction discourages some students from completing the 

bridge courses Faculty identified technology literacy as being a challenge, both for older students who 
have limited experience using a computer and for students coming out of high school where technology 
access may not have been satisfactory. One faculty member characterized some older students as being 
“terrified” when they learned about the course approach. 
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Diagnostic data have not been implemented into the college data warehouse. 

The diagnostic assessment data also cannot be used to generate reports across departments or the 
college overall. They can only generate reports on an individual-student basis. Any analysis and 
reporting beyond individual students must be done manually. 

SPC is challenged to make bridge courses with early exit and transition to college-level courses 

work within the current credit hour and financial aid framework. Administrators identified 
working closely with the financial aid office to ensure that the My Bridge model with the college-level 
transition would work for students while also protecting their financial aid. Some instructors seemed 
unaware of how to move a student from one course into another course mid-semester if the student had 
completed all of the requirements of the first course early. 

Faculty members were concerned about the idea of broadening the effort beyond those who 

tested close to the placement cut-of They seemed to feel that students needed to be self-motivated 
to benefit from the modular, self-paced approach. They felt that they would lose lots of students who 
lacked the self-discipline to stay on top of their work. 

There is still a need for some level of traditional education. Students appreciated the weekly in- 
class meetings. They felt that this time was important because it allowed them to ask specific questions 
and get general instruction, and it held them accountable. They were unsure about the idea of moving 
the courses entirely online. 

Conclusions 

The site visit to SPC provided an example of how a college has implemented placement exams, 
diagnostic assessments, and modularized course work for its entire developmental education program. 
Although SPC has not implemented the P.E.R.T. Diagnostics in the way they were intended, the vendors 
it has selected have designed their products around the P.E.R.T. competencies. While students, faculty, 
and administrators openly shared some challenges moving forward, the overall feeling around SPC’s 
developmental education and the My Bridge Program was very positive. Students expressed clear 
approval of the modularized courses, and faculty echoed the students in sharing that the courses and 
the online tools were effective for instruction. As the administrators identified at the onset of the visit, 
SPC has more work to do to design qualitative and quantitative assessment models to evaluate the 
program’s effectiveness, making the linking of diagnostic data to institutional student records even 
more imperative. 


26 


RESEARCH for ACTION 


Developmental Education and Diagnostic Assessment Reform 
Community College Case Study: Tallahassee Community College (TCC) 


The following profile has been developed to outline the 
local context and implementation of developmental 
education and diagnostic assessment reform at Tallahassee 
Community College. The profile was verified by the point of 
contact at the college; data included in the profile offers a 
stand-alone summary and will be used for cross-site 
analyses. The profile addresses six areas: l) local and 
community college context, 2) overview of the research, 

3) integration of placement and diagnostic assessment, 

4) placement and diagnostic assessment data use, 

5) perceptions of the effectiveness of the assessment 
system and developmental education program, and 6) 
challenges to implementation. The following research 
questions guided the site visits at the community college: 

• How do external calls for assessment and curricular 
alignment (e.g., state postsecondary readiness 
standards) affect institutional offerings? 

• What is the role of additional training and support— 
from states, coordinating boards, K-12, workforce 
partners, and other stakeholders— provided to 
institutions? 

• How tightly coupled are institutional programs and 
diagnostic assessments? In what ways do institutions 
incorporate diagnostic assessment data in student-level 
data systems? 

• Do implementation challenges differ when state- 
developed diagnostic tests are used versus those 
purchased from national vendors? Along what 
dimensions, and why? 

• What are the perspectives of administration, faculty, 
and students concerning the effectiveness of diagnostic 
assessment systems? What factors do institution 
stakeholders cite as barriers to successful 
implementation? 


Florida State Context 

In order to measure student skills against 
the state’s Postsecondary Readiness 
Competencies (PRCs) as they enter the 
Florida College System, the Postsecondary 
Education Readiness Test (P.E.R.T.) system 
was developed as Florida’s new 
“preferred” placement test. Florida 
College System institutions began 
administering the P.E.R.T. in October 
2010. In addition to P.E.R.T. Placement, the 
P.E.R.T. Diagnostic is another component 
of the assessment system. While the 
placement test determines appropriate 
course assignments, the P.E.R.T. Diagnostic 
identifies, by competency, where skill 
deficiencies exist. The P.E.R.T. Diagnostic 
has been developed to align with the 
Developmental Education Competencies 
that are the basis for developmental 
education courses offered at all Florida 
College System institutions. The goal of 
the P.E.R.T. Diagnostic is to allow faculty 
to target the areas in which students need 
additional work. The P.E.R.T. Diagnostic is 
voluntary for Florida colleges and is not 
computer-adaptive, so that all students are 
assessed on the same content as questions 
do not change based on previous 
responses.. Optional developmental 
education course modules have also been 
developed by the Florida Department of 
Education. 
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I. Local and Community College Context 

Local Context 

Tallahassee is located in the Florida panhandle, is the state capital, and is the location of Florida State 
University and Florida A&M University. The State of Florida and educational institutions, including 
Tallahassee Community College, are the primary employers in the city. Table l shows that the 
population is over 180,000, primarily white (57.4%) and African American (35%), and that the vast 
majority of residents have high school diplomas, but fewer than half have four-year degrees. 

Table I: Populations and Demographics 


POPULATION 

Total Population 

182,965 

Percentage of Population with High School Diploma 

91.6% 

Percentage of Population with Four-Year Degree 

46.5% 

Average Household Income 

$38,972 

DEMOGRAPHICS 

Percentage White 

57.4% 

Percentage African American 

35% 

Percentage Hispanic 

6.3% 

Percentage Asian 

3.7% 

Percentage Other 

2.6% 


Source: U.S Census Bureau State & County Quick Facts at http://quickfacts.census.gov/qfd/states/ 1 2/ 1 270600.html 


Community College Context 

Tallahassee Community College was opened in 1966 and currently has one main campus along with five 
centers. The college serves the postsecondary education needs of students from a district comprised of 
Gadsden, Leon, and Wakulla Counties. The majority of students apply for Associate of Arts degrees 
(73%) or Associate in Science or Applied Science degrees (13%) with just over one percent seeking a 
certificate; nearly 75 percent of Associate of Arts graduates go on to study at state universities in 
Florida. TCC is a large campus with over 20,000 students and over 400 instructional FTE staff. TCC is 
an Achieving the Dream Leader College and has focused on interventions in developmental education, 
including new, mandatory student orientation and a revamped academic advising program since 2007. 
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Table 2: Student Characteristics and Academic Programs 


STUDENT CHARACTERISTICS 

Unduplicated Headcount 

21,243 

Total Full-Time Equivalent (FTE) Enrollment 

12,219 

Percentage White Students 

49% 

Percentage African-American Students 

34% 

Percentage Hispanic Students 

9% 

Percentage Other Students 

8% 

Full-Time Beginning Undergraduates Receiving Financial Aid 

82% 4 

ACADEMIC PROGRAMS* 

Number of Bachelor’s Degree Programs (and Graduates) 

0 

Number of Associate Degree Programs (and Graduates) 

31 (3,298) 

Number of Certificate Programs (and Graduates) 

57 (489) 

Number of Diploma Programs (and Graduates) 

0 


Source: Tallahassee Community College and other sources as referenced (see footnotes) 


II. Overview of the Research 

Research for Action (RFA) conducted a field work visit (see Table 3) in the spring semester of 2013 to 
explore the implementation and impact of the Assessment and Learning in Knowledge Spaces (ALEKS) 
program and the P.E.R.T. Placement and P.E.R.T. Diagnostic assessments on math developmental 
education instruction, and to compare student experiences at the college with each. 


Table 3: Research Conducted 

DATA SOURCES 

Interviews with Administrators 2 

Interviews with Developmental Education Math Faculty 3 

Math Developmental Education Faculty Focus Groups I 

Math Developmental Education Student Focus Groups I 


In addition, we conducted background research using institutional websites, documents, and phone and 
email conversations with college administrators. 


4 Integrated Postsecondary Education Data System (IPEDS) Data Center at http://nces.ed.gov/ipeds/datacenter/ 
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III. Integration of Placement and Diagnostic Assessment at the Community College 

All three of the assessment instruments (P.E.R.T. Placement, P.E.R.T. Diagnostic, and ALEKS) have 
been used to varying degrees across the college, and are discussed below (see Table 4): 


Key for Table 4 through Table 6 

O YES 

( 

8 

) NO 


Table 4: Level of Diagnostic Assessment Integration within the Community College 


STATE-DEVELOPED DIAGNOSTIC ASSESSMENT (P.E.R.T. DIAGNOSTIC) 

Administered in Math? 

O 

Administered in English? 

® 

Fully Implemented with all Students? 


VENDOR-DEVELOPED DIAGNOSTIC ASSESSMENT 

Administered at the Community College? 

O 

Administered in Math? 

O 

Administered in English? 

N/A 

Fully Implemented with all Students? 

O 


Integration across Subjects and Students: The focus of the site visit was on the comparison between 
the P.E.R.T. and ALEKS diagnostic assessments as they are both used in math; additional diagnostic 
tests and curricular tools are also used in developmental English. 

- P.E.R.T.: The placement test is administered campus-wide to all students as they enter the 
college. The diagnostic test has been field-tested in a subset of accelerated developmental 
education math courses. Separate from the P.E.R.T. Placement test, the P.E.R.T. Diagnostic test 
and the ALEKS diagnostic test were both administered at the beginning of the math bridge 
program, a two-week course for students identified by the P.E.R.T. Placement test as being very 
close to eligible for credit-bearing math coursework. The use of the P.E.R.T. Diagnostic was part 
of a grant program funded by the Florida Department of Education to encourage the use of the 
diagnostic test as part of one- and two-credit developmental education classes aimed at 
decreasing the time it takes for students to finish developmental education and enroll in credit- 
bearing courses. 

- Vendor-Developed Assessments and Software Tools: TCC has been using the ALEKS diagnostic 
test and online modules across courses in the math developmental education program for at 
least five years. ALEKS is a web-based assessment and learning system that uses adaptive 
questioning to determine what skills each student knows and does not know in developmental 
math content. The ALEKS diagnostic assessment is given once students are placed in a 
developmental education math course (0018 or 0028) to determine the areas in which students 
need additional instruction through the ALEKS online instructional modules, which are 
completed outside of traditional classroom instructional time. 
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Impact on College Policy and Practice: The use of the P.E.R.T. Placement and P.E.R.T. Diagnostic 
have become common but have not influenced college policy overall. When asked about the impact of 
the P.E.R.T. Placement and P.E.R.T. Diagnostic tests on college policy and practice, administrators 
stated that they had not changed any major policy at the college. The P.E.R.T. Placement test has simply 
replaced the Accuplacer assessment system that had been used prior to the state initiative, and the 
P.E.R.T. Diagnostic has only been used with a small cohort of students to date. The use of the ALEKS 
diagnostic test over a period of years, and the success faculty have found with ALEKS overall, have led 
them to believe that they may continue to use it as opposed to shifting to the P.E.R.T. Diagnostic. 

Impact on Instruction Data from the ALEKS diagnostic test determines the areas of online tutoring 
modules, but does not influence traditional classroom instruction. The developmental education 
curriculum in math has been standardized by the state department of education and requires the faculty 
to stay focused on covering the course content, and does not allow for differentiation. One faculty 
member explained that “we have to push every class to get through all of the material; it is brutal.” 


IV. Assessment Data Use 

The use of diagnostic test data from both the P.E.R.T Diagnostic and other, vendor-developed 
diagnostic tests is outlined in Table 5 below and explores the extent to which the data informs 
developmental education as well as other college programs. As mentioned previously, the focus of the 
site visit was the math diagnostic assessment. 


Table 5: Use of Diagnostic Assessment Data 


STATE-DEVELOPED DIAGNOSTIC ASSESSMENT (P.E.R.T. DIAGNOSTIC) 

MATH 

ENGLISH 

Are the diagnostic test results included in the college student data system? 

® 

® 

Is the test providing data that can be used to identify individual student 
learning needs? 

O 

O 

Are the test data linked to other student records? 

® 

N/A 

Are the data being used outside of student placement and diagnostic 
decisions in developmental education? 

® 

N/A 

VENDOR-DEVELOPED DIAGNOSTIC ASSESSMENT 



Are the diagnostic test results included in the college student data system? 

® 

® 

Is the test providing data that can be used to identify individual student 
learning needs? 

O 

N/A 

Are the test data linked to other student records? 

® 

N/A 

Are the data being used outside of student placement and diagnostic 
decisions in developmental education? 

® 

N/A 


Inclusion in the Student Data System: Diagnostic data is not incorporated into the college data 
warehouse, despite the widespread use of diagnostic assessment data in TCC’s developmental education 
program. 


Identification of Individual Student Learning Needs: Both the P.E.R.T. Diagnostic and the ALEKS 
diagnostic assessment produce individual student data broken down by subtopic and/or question so 
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faculty and students can see exactly where there are needs for additional instruction. 

Linkage to Other Student Data: Diagnostic data is not incorporated in data reports on other student 
characteristics. Further, ALEKS and P.E.R.T. diagnostic assessment data has not been used to generate 
reports across departments or the college overall, but only generates reports on an individual student 
basis. 

Data Use outside of Developmental Education: Diagnostic data is not used outside of the 
developmental education program. In part because ALEKS and P.E.R.T. diagnostic assessment data 
cannot be used to generate reports across departments or the college overall, but only generates reports 
on an individual student basis, the data has not been used for larger college evaluation. Developmental 
education faculty members compile individual student data to determine program progress, but the 
process is cumbersome and has not allowed for wider data use. 

V. Effectiveness of the Assessment System and Developmental Education Program 


The reported level of effectiveness varied among the placement assessments and curricular tools used at 
TCC (see Table 6): 

Table 6: Respondent Perceptions of the Effectiveness of Assessments and Curriculum Redesign 


STATE-DEVELOPED ASSESSMENT (P.E.R.T.) 

MATH 

ENGLISH 

Is the test placing students more accurately in the developmental education 
sequence? 

® 

® 

Is the curriculum redesign allowing students to complete the developmental 
education sequence more quickly? 

N/A 

N/A 

VENDOR-DEVELOPED DIAGNOSTIC ASSESSMENT 



Is the test placing students more accurately in the developmental education 
sequence? 

O 

N/A 

Is the curriculum redesign allowing students to complete the developmental 
education sequence more quickly? 

O 

N/A 


Student Placement: 

- Math: The P.E.R.T. Placement test has had little impact on developmental education math 
placement. Perhaps because of the similar content covered in math developmental education 
programs, the college has seen little change in the way students are placed in developmental 
math courses as a result of the switch from the Accuplacer to the P.E.R.T. Placement test. 
However, the rigor of the test as compared to the ALEKS has been an issue based on the 
experience of students in the bridge program. Students placed in the program have been 
determined to be only a few points away from eligibility for credit-bearing coursework in math 
according to the P.E.R.T., but have commonly been found to need considerable remediation 
across a large portion of topics based on the ALEKS diagnostic test. One faculty member 
explained, “the P.E.R.T. is about broader competencies and the ALEKS is more fine-grained,” 
while another said that “the P.E.R.T. does not assess all of the individualized skills that that 
ALEKS covers.” 
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English: In contrast, placement in reading has changed dramatically as a result of the P.E.R.T. 
Placement test. The P.E.R.T. Placement tests in reading have placed many more students in 
credit -bearing courses than the Accuplacer did previously. Consequently, the developmental 
education program in English has lost multiple sections. As a result, there has been talk around 
the state regarding the changes in results, and adjustments have been made to the cut-off scores. 
It is too early to tell how well students recently placed in credit-bearing English courses based 
on the P.E.R.T. Placement test are faring as compared to those placed by the Accuplacer. 

Diagnostic Assessment: The P.E.R.T. Diagnostic assessment has been used with only a small cohort of 
students, but faculty respondents’ early impressions are that the test is less effective than the ALEKS 
diagnostic test used more widely at TCC. While the P.E.R.T. is aligned to the state postsecondary 
readiness competencies and reports on how the students score on each of those competencies, the 
ALEKS provides more detail about what students know, and translates “seamlessly” into the online 
curricula students use to address the areas in which they need additional instruction. The connection 
between the test and the online instructional tools makes the test data more actionable than data from 
the P.E.R.T. Diagnostic. 

Developmental Course Completion: There was consensus across respondents that the ALEKS 
program has been effective in moving students through developmental education courses in math more 
quickly than before it was in place. Faculty and administrators also reported that the data clearly show 
that students who complete developmental education with ALEKS are stronger in their fundamental 
math skills than students who do not. 

Supports to Increase Effectiveness 

McGraw-Hill has worked with TCC to customize the ALEKS program to fit the college’s needs. 
With the development of the state’s Postsecondaiy Readiness Competencies (PRCs) and the 
current expectations to modularize instruction, McGraw-Hill has worked with TCC over time to 
adapt the ALEKS to meet its changing standards and curricular reforms. Most recently, a team 
from TCC met with McGraw-Hill staff to restructure the way in which the ALEKS math skills are 
organized to align with the new course modules TCC is developing to meet the state’s 
expectations for developmental education reform. 

- College administrators and faculty provide annual training on the use of ALEKS. Each fall, 

developmental math faculty members are provided with training from the program chair on how 
to implement the ALEKS modules and diagnostic assessment. Faculty reported that they were 
familiar with the ALEKS and found it accessible to students. 

VI. Challenges to Implementation 

A number of issues were raised as barriers to implementation of the P.E.R.T. Placement and Diagnostic 
systems: 

There has been a lack of emphasis from the state on implementing the P.E.R.T. tests. Both the 
placement and diagnostic versions of the new assessment are voluntary in the state. Information about 
the assessments from the Florida Department of Education was described by faculty and administrators 
as limited. 
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TCC has a history of implementation with other diagnostic assessments. The college has been 
using the ALEKS program in math, as well as Plato and MyCompLab in reading and writing for a 
number of years. Each of these not only provides a diagnostic assessment, but also provides interactive 
online instruction, monitoring and tracking capabilities, progress reports, and feedback that a 
diagnostic assessment alone does not provide. 

Diagnostic data use has been a challenge. While the P.E.R.T. Placement test data is housed in the 
college’s data warehouse, diagnostic data is not housed centrally and “are very disconnected” from other 
data sources. The ALEKS and P.E.R.T. diagnostic assessment data also cannot be used to generate 
reports across departments or the college overall, but only generates reports on an individual student 
basis. 

The P.E.R.T. Diagnostic test takes longer to administer than the ALEKS test. Faculty reported 
that the P.E.R.T. Diagnostic test takes on average about 15 minutes longer to administer than the 
ALEKS test. With the P.E.R.T. Diagnostics separate from P.E.R.T. Placement test, the extra time it takes 
to administer becomes a disincentive for implementation. 

Students are often unprepared to take the P.E.R.T. Placement test. Despite the development at 
TCC of a mandatory orientation program supported by Achieving the Dream, students are often 
unaware that they are required to take the P.E.R.T. Placement test until the day when it is administered 
to them. While review materials and a practice test are available to students, faculty and administration 
respondents reported that few students use these resources prior to taking the test. 


Conclusions 

The site visit to TCC allowed for the comparison of an “off the shelf’ vendor-developed diagnostic 
assessment and developmental math instructional tool (ALEKS) with a state-developed diagnostic test 
aligned to the state’s postsecondary standards (P.E.R.T.). With the college’s history of successful 
implementation of the ALEKS and the vendor’s willingness to customize their product to fit the needs of 
the college in response to state requirements, the P.E.R.T. Diagnostic was seen as “redundant” in many 
ways. The college’s experience with the P.E.R.T. Placement test has been mixed: results were similar to 
the previous system in math placement, but were troubling in reading due to the sharp increase in the 
percentage of students being placed in credit -bearing courses, which has led to concerns about the rigor 
of the test and adjustments to the cut-off scores in the state. The diagnostic data from both the P.E.R.T. 
Diagnostic and ALEKS have not been integrated into the college’s larger data warehouse, and have 
therefore not been used to inform larger issues at TCC. Overall, the P.E.R.T. Placement and Diagnostic 
tests have not proved game changers at the college or increased the existing efficiency in developmental 
education. 
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RESEARCH for ACTION 


Developmental Education and Diagnostic Assessment Reform 
Community College Case Study: Valencia College 


The following profile has been developed to outline the 
local context and implementation of developmental 
education and diagnostic assessment reform at Valencia 
College. The profile was verified by the point of contact at 
the college; data included in the profile offers a stand-alone 
summary and will be used for cross-site analyses. The 
profile addresses six areas: l) local and community college 
context, 2) overview of the research, 3) integration of 
placement and diagnostic assessment, 4) placement and 
diagnostic assessment data use, 5) perceptions of the 
effectiveness of the assessment system and developmental 
education program, and 6) challenges to implementation. 
The following research questions guided the site visits at 
the community college: 

• How do external calls for assessment and curricular 
alignment (e.gf., state postsecondary readiness 
standards) affect institutional offerings? 

• What is the role of additional training and support— 
from states, coordinating boards, K-12, workforce 
partners, and other stakeholders— provided to 
institutions? 

• How tightly coupled are institutional programs and 
diagnostic assessments? In what ways do institutions 
incorporate diagnostic assessment data in student-level 
data systems? 

• Do implementation challenges differ when state- 
developed diagnostic tests are used versus those 
purchased from national vendors? Along what 
dimensions, and why? 

• What are the perspectives of administration, faculty, 
and students concerning the effectiveness of diagnostic 
assessment systems? What factors do institution 
stakeholders cite as barriers to successful 
implementation? 


Florida State Context 

In order to measure student skills against 
the state’s Postsecondary Readiness 
Competencies (PRCs) as they enter the 
Florida College System, the Postsecondary 
Education Readiness Test (P.E.R.T.) system 
was developed as Florida’s new 
“preferred” placement test. Florida 
College System institutions began 
administering the P.E.R.T. in October 
2010. In addition to P.E.R.T. Placement, the 
P.E.R.T. Diagnostic is another component 
of the assessment system. While the 
placement test determines appropriate 
course assignments, the P.E.R.T. Diagnostic 
identifies, by competency, where skill 
deficiencies exist. The P.E.R.T. Diagnostic 
has been developed to align with the 
Developmental Education Competencies 
that are the basis for developmental 
education courses offered at all Florida 
College System institutions. The goal of 
the P.E.R.T. Diagnostic is to allow faculty 
to target the areas in which students need 
additional work. The P.E.R.T. Diagnostic is 
voluntary for Florida colleges and is not 
computer-adaptive, so that all students are 
assessed on the same content as questions 
do not change based on previous 
responses. . Optional developmental 
education course modules have also been 
developed by the Florida Department of 
Education. 
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I. Local and Community College Context 

Local Context 

Valencia has multiple campuses in the Orlando area, which is home to some of the nation’s most 
famous tourist attractions such as Walt Disney World, Universal Studios, and SeaWorld. Orlando is also 
home to the University of Central Florida and the third-largest metro area in the state of Florida. The 
largest source of employment in the Orlando area is the tourism and entertainment industry, with Walt 
Disney World Resorts being the largest individual employer. Table l shows that the population is over 
240,000, primarily white (41.3%) but with substantial populations of African Americans (28.1%) and 
Hispanics (25.4%). Although the vast majority of residents have high school diplomas, less than a third 
of the population has a four-year degree or higher. 

Table I: Orlando Area Populations and Demographics 


POPULATION 

Total Population 

243,195 

Percentage of Population with High School Diploma 

86.7% 

Percentage of Population with Four-Year Degree 

31.9% 

Average Household Income 

$42,755 

DEMOGRAPHICS 

Percentage White 

41.3% 

Percentage African American 

28.1% 

Percentage Hispanic 

25.4% 

Percentage Asian 

3.8% 

Percentage Other 

3.9% 


Source: U.S Census Bureau State & County Quick Facts at http://quickfacts.census.gov/qfd/states/ 1 2/ 1 253000.html 

Community College Context 

Valencia Junior College was opened in the fall of 1967, in a few small portable buildings. The name was 
changed to Valencia Community College in 1971, and then renamed again to Valencia College in 
December of 2010 when the institution’s mission was expanded to allow the offering of bachelor’s 
degrees. The campus currently includes five campuses (West, East, Osceola, Winter Park, and Lake 
Nova), one center (Downtown Center), and one institute (Criminal Justice Institute). The majority of 
students come from Orange County, with significant numbers coming from Osceola County. 

The majority of students apply for Associate of Arts degrees (54%) or technical certificates (31.9%). 
Associate in Science or Associate in Applied Science degrees make up a mere 11.8%, followed by career 
and EPI certificates at 2.3%. Of those who earn an Associate of Arts degree and continue their studies in 
the state college system, over 89% do so at the University of Central Florida. 

Valencia enrolls almost 65,000 students annually and employs over 480 full-time faculty members. 
Valencia is an Achieving the Dream Leader College and has focused on retention rates. They utilized the 
Developmental Education Redesign and Modularization Mini-Grant offered by the Florida College 
System to initiate new interventions in developmental education. 
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Table 2: Student Characteristics and Academic Programs 


STUDENT CHARACTERISTICS 

Unduplicated Headcount 

64,912 

Total Full-Time Equivalent (FTE) Enrollment 

32,256 

Percentage White Students 

36% 

Percentage African-American Students 

17% 

Percentage Hispanic Students 

30.5% 

Percentage Other Students 

16.2% 

Full-Time Beginning Undergraduates Receiving Financial Aid 

85% 

ACADEMIC PROGRAMS 

Number of Bachelor’s Degree Programs (and Graduates) 

2(0) 

Number of Associate Degree Programs (and Graduates) 

36 (7,810) 

Number of Certificate Programs (and Graduates) 

85 (4,062) 

Number of Diploma Programs (and Graduates) 

0 


Source: Valencia College 


II. Overview of the Research 

Research for Action (RFA) contracted with three representatives from Florida State University to 
conduct field work visits (see Table 3) in the spring semester of 2013. The field work explored the 
implementation and impact of the Foundations of Math program, the P.E.R.T. Placement exam, and the 
use of diagnostic assessments on math developmental education. Site visits were conducted at the West 
Campus of Valencia College. 

Table 3: Research Conducted 

DATA SOURCES 

Interviews with Administrators I 

Interviews with Developmental Education Math Faculty I 

Math Developmental Education Student Focus Groups I 

In addition, we conducted background research using institutional websites, documents, and phone and 
email conversations with college administrators. 

III. Integration of Placement and Diagnostic Assessment at the Community College 

Table 4 summarizes where and how the diagnostic testing is utilized and integrated into Valencia 
College’s system. The use of placement tests, diagnostic assessments, and developmental math courses 
is discussed within the context of the West Campus of Valencia. Valencia uses the P.E.R.T. upper level 
math diagnostic test with the modularized developmental math courses for those who scored in the 110 
to 112 range on the P.E.R.T. Placement exam in math, indicating the student falls just short of the 
ability to enter college credit -bearing math courses. Students who select this modularized approach 
utilize the Pearson Foundations of Math software, which allowed the state competencies to be 
integrated into the modules. 
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Key for Table 4 through Table 6 

O YES 

( 

8 ) 

NO 


Table 4: Level of Diagnostic Assessment Integration within the Community College 


STATE-DEVELOPED DIAGNOSTIC ASSESSMENT (P.E.R.T. DIAGNOSTIC) 

Administered in Math? 

O 

Administered in English? 

® 

Fully Implemented with All Students? 

®. 

VENDOR-DEVELOPED DIAGNOSTIC ASSESSMENT 

Administered at the Community College? 

® 

Administered in Math? 

N/A 

Administered in English? 

N/A 

Fully Implemented with All Students? 

N/A 


*Only students near the cut-off for developmental education are able to self-select into the modular course, which includes the 
diagnostic. 

Integration across Subjects and Students: 

- P.E.R.T.: 

- Placement: The placement tests are implemented campus-wide. Valencia requires all 
students to take the P.E.R.T. Placement test unless they can provide other approved test 
scores or transcripts demonstrating college-level English or mathematics coursework with a 
grade of “C” or higher from a regionally accredited institution. The P.E.R.T. Placement is the 
preferred, but not required, placement test in Florida, and replaced the Accuplacer test that 
was used previously. 

- Diagnostic: The P.E.R.T. Upper Level Math Diagnostic test is then offered to students who 
have scored within the range of no to 112 on the P.E.R.T. in math. The P.E.R.T. Diagnostic 
was implemented as part of the grant program funded by the Florida Department of 
Education to encourage the use of the diagnostic test as part of one- and two-credit 
developmental education classes aimed at decreasing the time it takes for students to finish 
developmental education and enroll in credit -bearing courses. 

- Vendor-Developed Assessments and Software Tools: The Pearson Foundations of Math 
software was selected because the college had been using the software for some time, and 
because it included an option that would allow the state competency standards to be matched to 
the modules. Although it is a separate system from the P.E.R.T. Diagnostic test, the results of the 
Diagnostic are entered into the Pearson software to assign specified modules to students based 
on their diagnostic results. Foundations of Math is a web-based learning system that uses self- 
paced modules covering: (1) exponents and polynomials, (2) factoring, (3) linear equations, (4) 
radicals, and (5) graphing. 
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Impact on College Policy and Practice: 

- P.E.R.T. Placement and Diagnostic Tests: While the tests have become commonly used, they 
have not brought about substantial changes to college policy. Prior to the implementation of the 
P.E.R.T. Placement tests two years ago, the Accuplacer was the college’s placement test of 
choice, with options for exemption which still included the SAT or ACT exams. The state’s 
initiatives, and especially the Developmental Education Redesign and Modularization Mini- 
Grant, propelled the implementation of the P.E.R.T. Placement and P.E.R.T. Diagnostic. 

- Pearson Foundations of Math: The software was selected because the college had been using it 
for some time and it included an option that would allow the state competency standards to be 
matched to the modules. 

Impact on Instruction: The Pearson module delivery system has influenced the delivery of classroom 
instruction, but only for the early developmental courses. The expected student competencies have been 
standardized by the Florida Department of Education. To ensure that students have what they need for 
entrance into credit -bearing courses, the diagnostics combined with modular delivery has allowed 
instructors to facilitate accelerated but self-paced instruction for students. However, this mode of 
delivery is only specific to the two developmental math courses. 

IV. Assessment Data Use 

The use of diagnostic test data from both the P.E.R.T Diagnostic and other, vendor-developed 
diagnostic tests is outlined in Table 5 below and explores the extent to which the data informs 
developmental education as well as other college programs. As mentioned previously, the focus of the 
site visit was the math diagnostic assessment. 


Table 5: Use of Diagnostic Assessment Data 


STATE-DEVELOPED DIAGNOSTIC ASSESSMENT (P.E.R.T. DIAGNOSTIC) 

MATH 

ENGLISH 

Are the diagnostic test results included in the college student data system? 

® 

N/A 

Is the test providing data that can be used to identify individual student 
learning needs? 

O 

N/A 

Are the test data linked to other student records? 

® 

N/A 

Are the data being used outside of student placement and diagnostic 
decisions in developmental education? 

® 

N/A 

VENDOR-DEVELOPED DIAGNOSTIC ASSESSMENT 



Are the diagnostic test results included in the college student data system? 

N/A 

N/A 

Is the test providing data that can be used to identify individual student 
learning needs? 

N/A 

N/A 

Are the test data linked to other student records? 

N/A 

N/A 

Are the data being used outside of student placement and diagnostic 
decisions in developmental education? 

N/A 

N/A 
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Inclusion in the Student Data System: P.E.R.T. Placement and Diagnostic data are not incorporated 
into the college data system. All the current placement and diagnostic data are housed within one office 
and available to faculty as needed. Although the administration indicated that P.E.R.T. Placement and 
Diagnostic data can be integrated via their institutional research department, and that this was done in 
the past with CPT and Accuplacer, the current system has not been integrated to include the new 
P.E.R.T. results. 

Identification of Individual Student Learning Needs: The P.E.R.T. Diagnostic reports student test 
results for each of the questions included on the test so that faculty can see where students are 
struggling. 

Linkage to other Student Data The implementation of the diagnostic tool has been recent, and the 
diagnostic data are not incorporated into the school-wide data systems, so the use of the diagnostic data 
has been limited to the math program and is not linked to other student data. 

Data use outside of Developmental Education: Diagnostic data are not used outside of the 
developmental education program. There has been discussion about widening its use to help assign 
more or less independent work for students based on diagnostic scores. The discussions have not gone 
beyond course design. 


V. Effectiveness of the Assessment System and Developmental Education Program 

Table 6, provided below, summarizes findings of effectiveness and accuracy of assessment and 
diagnostic testing results as it relates to Valencia. 


Table 6: Respondent Perceptions of the Effectiveness of Assessments and Curriculum Redesign 


STATE-DEVELOPED ASSESSMENT (P.E.R.T.) 

MATH 

ENGLISH 

Is the test placing students more accurately in the developmental education 
sequence? 

Unknown 

N/A 

Is the curriculum redesign allowing students to complete the developmental 
education sequence more quickly? 

N/A 

N/A 

VENDOR-DEVELOPED DIAGNOSTIC ASSESSMENT 



Is the test placing students more accurately in the developmental education 
sequence? 

N/A 

N/A 

Is the curriculum redesign allowing students to complete the developmental 
education sequence more quickly? 

O 

N/A 


Student Placement The respondents gave no indication of any belief that the P.E.R.T. Placement was 
more or less accurate than the prior placement test (Accuplacer), at least outside of the increased 
duration required for administering the test. Because the implementation of the P.E.R.T. Placement is 
so recent, there is not enough information yet to determine if the test is placing students correctly. 
However, the Director of Standardized Testing described the P.E.R.T. Placement as very similar to 
other placement tools, with the exception of length. 


40 




Diagnostic Assessment Students that were interviewed felt that the P.E.R.T. Diagnostic adequately 
gauged their needs but was very long and time-consuming to take. Faculty also agreed that the 
diagnostic test does an effective job of identifying student needs. However, the number of students who 
have entered the accelerated course options has been so small that results are preliminary. The faculty 
and administration felt that the program would be of much higher quality if the diagnostic test were 
more fully integrated with the state competencies and into the modular software. As of now, the 
connection between the diagnostic test and the modules is not seamless. 

Developmental Course Completion; Students and faculty felt that the mix of online and 
modularized course delivery through the Pearson Foundations of Math online instructional modules, 
along with set class time where an instructor was available, was a very beneficial mix. It allowed 
students to pace themselves, and instructors to monitor how much time students were applying 
themselves to completing the course materials. Additionally, because the Pearson software does not 
provide the flexibility to integrate additional activities or information resources, the faculty members 
are able to use the in-class time to provide those tools. Students are able to ask the instructors for 
assistance in completing mathematical procedures that they did not understand from the software 
instruction. Students felt that the software motivated them to stay on top of their work, and preferred 
the modular approach over the traditional in-person, classroom-based approach. 

Supports to Increase Effectiveness: Pearson trained faculty members in the use of the Foundations of 
Math software and in methods for integrating the P.E.R.T. Diagnostic results into the assignment of 
modules for students. The faculty and administration indicated that they had a long-standing 
relationship with Pearson, so it was natural for them to implement the software and integrate it with the 
use of the P.E.R.T. Diagnostic. Although the P.E.R.T. Diagnostic, developed by McCann, was not 
directly connected with the Pearson product, they were given some foundational instruction on how to 
proceed. 


VI. Challenges to Implementation 

A number of barriers and challenges to implementation of the P.E.R.T. Placement and Diagnostic 
Systems were discussed: 

Funding was not initially allocated for the P.E.R.T. Diagnostic. Funding was initially allocated 
toward the P.E.R.T. Placement for every student to take the test once. Additional retakes were allowed 
once per year at a cost of $10 per sub-test, but the student could take the first one without a fee. In 
contrast, when the P.E.R.T. Diagnostic was implemented, the cost per student to take the Diagnostic 
was not included. The administration agreed to use their budget surplus to cover the cost per student 
until the grant funding kicked in to “reimburse” the testing center for the costs of the diagnostic. 

The P.E.R.T. Diagnostic and the state competencies were not integrated and did not use the 

same coding Faculty members had to take a significant amount of time to read each of the diagnostic 
descriptions and each of the state competency descriptions and attempt to match them up. 
Furthermore, one of the competencies was missing from the diagnostic (rational expressions), which 
forced all developmental education math students to take the associated module. The faculty felt that 
this content is important and should have been addressed. 
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There was limited information sharing from the state on the P.E.R.T. tests. Both the placement 
and diagnostic versions of the new assessment, although voluntary in Florida, were highly encouraged 
through the grant. But the information about the assessments from the Florida Department of 
Education was limited. Furthermore, the amount of time available for transitioning from grant 
application to full implementation was short, and the administration and faculty felt very rushed in the 
process. 

The P.E.R.T. Placement and Diagnostic tests require a substantial amount of time to take. There 
are limited testing labs on campus, and when combined with the length of time needed to complete the 
P.E.R.T. Placement and Diagnostic, the infrastructure is not ready to handle a scale-up of the diagnostic 
testing and new modular delivery of courses. The Director of Standardized Testing was also concerned 
with the tests contributing to testing fatigue. 

Students are unaware of the options available to them. The current system of recruiting students 
requires a great time commitment from the administration. Students are selected based on their scores 
in the P.E.R.T. Placement test. These students are then emailed (followed up by individual phone calls) 
to inform them of the option to take the accelerated courses after the taking of the additional P.E.R.T. 
Diagnostic. Because the initial mode of contact was email, which students may not have gotten or may 
have left unread, many seemed to be unaware of the option until the Dean of Mathematics called the 
students directly. Even then, few opted to enroll in the course. 


Conclusions 

This site visit to Valencia College provided an example of how a college has taken the state- developed 
diagnostic test aligned to the state postsecondary standards (P.E.R.T.) and used that information to 
assign modules of an “off the shelf’ developmental math instructional tool (Pearson’s Foundations of 
Math). The resulting effects on the first few students to take the modularized courses after taking the 
diagnostic assessment seem to be very positive, with the students passing a post-instruction test with 
exemplary scores. These results, however, are based on only five students who have complete data, so 
they must also be taken with some reservation. Furthermore, until the data from the P.E.R.T. 
Placement and Diagnostic tests are fully integrated with the college’s data warehouse system, use of the 
results outside of the Developmental Math department is nonexistent, and the data are unavailable for 
use in studying wider issues at Valencia. 
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RESEARCH for ACTION 


Developmental Education and Diagnostic Assessment Reform 
Community College Case Study: Danville Community College (DCC) 

The following profile has been developed to outline the 
local context and implementation of developmental 
education and diagnostic assessment reform at Danville 
Community College. The profile was verified by the point of 
contact at the college; data included in the profile offer a 
stand-alone summary and will be used for cross-site 
analyses. The profile addresses six areas: l) local and 
community college context, 2) overview of the research, 3) 
integration of placement and diagnostic assessment, 4) 
placement and diagnostic assessment data use, 5) 
perceptions of the effectiveness of the assessment system 
and developmental education program, and 6) challenges 
to implementation. The following research questions 
guided the site visits at the community college: 

• How do external calls for assessment and curricular 
alignment (e.gf., state postsecondary readiness 
standards) affect institutional offerings? 

• What is the role of additional training and support— 
from states, coordinating boards, K-12, workforce 
partners, and other stakeholders— provided to 
institutions? 

• How tightly coupled are institutional programs and 
diagnostic assessments? In what ways do institutions 
incorporate diagnostic assessment data in student-level 
data systems? 

• Do implementation challenges differ when state- 
developed diagnostic tests are used versus those 
purchased from national vendors? Along what 
dimensions, and why? 

• What are the perspectives of administration, faculty, 
and students concerning the effectiveness of diagnostic 
assessment systems? What factors do institution 
stakeholders cite as barriers to successful implementation? 


Virginia State Context 

The Developmental Education Task Force 
(DETF) was convened in 2008 to review 
the system’s developmental education 
policies and increase student progress 
through their courses towards graduation; 
the initiative was supported by 
involvement in the Developmental Education 
Initiative. The DETF provided 
recommendations in the report The 
Turning Point: Developmental Education in 
Virginia’s Community Colleges the following 
year. In 2010, the Developmental 
Mathematics Redesign Team (DMRT) 
released recommendations for 
developmental mathematics courses across 
the Virginia Community College System 
(VCCS), proposing that the content of the 
developmental mathematics curriculum be 
revised, with different pathways contingent 
on a student’s program of study; that 
content be organized into nine pre-college 
units of study; and that the VCCS develop 
new placement and diagnostic instruments. 
The Developmental English Redesign Team 
(DERT) proposed that developmental 
English be restructured as an integrated 
reading and writing system, with three 
direct pathways to credit-bearing English 
determined by a student’s placement test 
score and performance against specific 
learning outcomes. 
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I. Local and Community College Context 
Local Context 

Danville is a moderately sized city near Virginia’s Southwestern border with North Carolina. With a 
population of over 140,000 among the three areas served by DCC, over three-quarters of the population 
has a high school diploma, while only 14 percent holds a four -year degree, less than half the rate in the 
state overall. The region is primarily white, while African Americans compose the second highest 
racial/ethnic group at just over a third (see Table 1). 

Table I : Aggregate Populations & Demographics for the City of Danville, Pittsylvania County, & Halifax County 

POPULATION 


Total Population 

Percentage of Population with High School Diploma 
Percentage of Population with Four-Year Degree 
Average Household Income 

141,652 

76.2% 

14.3% 

$35,971.36 

DEMOGRAPHICS 

Percentage White 

64.1% 

Percentage African American 

33.9% 

Percentage Hispanic 

2.3% 

Percentage Asian 

0.5% 


Source: U.S Census Bureau State & County Quick Facts at httD://quickfacts. census.gov/qfd/maDs/virginia map.html 


Economically, Danville has experienced a decrease of 18 percent in private employment (nonfarm) from 
2000-2012, in part due to the loss of tobacco and textile industry jobs in the area, resulting in more 
people looking to come back to school to receive job training. 5 However, the area has managed to attract 
new employment opportunities, such as Macerata Wheels LLC, which is going to move its plant to 
Danville and hire 100 people "due to its proximity to the machining and tool program at Danville 
Community College.” 6 


Community College Context 

Founded in 1966 through a merger of the Danville Technical Institute and the Danville Division of the 
Virginia Polytechnic Institute, DCC offers a mix of associate degree, diploma, and certificate granting 
programs. 7 Danville Community College has two campuses and serves the City of Danville and 
communities within Pittsylvania and Halifax Counties. Their main campus offers the college’s core 
services while the Regional Center for Applied Technology and Training focuses on technical and trade- 
based classes; 90 percent of graduates have been employed within trade industries (e.g. air conditioning 
and refrigeration, automotive analysis and repair, precision machining technology, etc.). 8 

Table 2, below, shows DCC’s 2011-12 enrollment numbers and academic programs offered. With nearly 
3,000 full time students, DCC is below the state average of about 5,600 in terms of the size of the 


5 U.S Census Bureau State & County QuickFacts. Retrieved from: http://quickfacts.ceiisus.gov/qfd/maps/virginia map.html 

6 Braun, A.D. (March, 2013). Custom wheel maker adding too jobs in Danville. The Business Journal. Retrieved from: 
httD://www.biziournals.com/triad/news/ 20 i 3 /o 3 /i 8 /custom-wheeTmaker-adding-ioo-iobs-in.html 

7 Danville Community College (2013). DCC program of study. Retrieved from: http : / / www. dec. vccs . edu /Academics / Programs .htm 

8 Danville Community College (2013). Fast facts. Retrieved from: http://www.dcc.vccs.edu/IR/ResearchFilesoq/FastFacts.pdf 
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student body. 9 Almost all students are white (57%) or African American (40%), with a few other ethnic 
groups represented (3%). 

Table 2: Student Characteristics and Academic Programs 


STUDENT CHARACTERISTICS 

Unduplicated Headcount 

6,493 

Total Full-Time Equivalent (FTE) Enrollment 

2,782 

Percentage White Students 

57% 

Percentage African-American Students 

40% 

Percentage Other Minority 

3% 

Full-Time Beginning Undergraduates Receiving Financial Aid 

79% 

ACADEMIC PROGRAMS 

Number of Bachelor’s Degree Programs (and Graduates) 

0 

Number of Associate Degree Programs (and Graduates) 

19 (353) 

Number of Certificate Programs (and Graduates) 

19 (452) 

Number of Diploma Programs (and Graduates) 

8 (82) 


Sources: Danville Community College Fast Facts; Virginia Community Colleges: Institutional Research Information 


DCC also has a number of state and national partners such as the Institute for Advanced Learning and 
Research, Southern Virginia Higher Education Center, and the University of Richmond. Since 2004, 
they have been working with Achieving the Dream (ATD) on initiatives focused on helping more 
community college students, particularly low-income students and students of color, to stay in school 
and earn a college certificate or degree 10 DCC has implemented a free two-week math bridge program 
targeted at first-year students; its curriculum helps students prepare for the Virginia Placement Test 
(VPT). As of fall 2012, the college saw dramatic increases in the movement of both minority and low- 
income students into higher levels of developmental math courses or directly into college-level math 
courses. 11 


II. Overview of the Research 

Research for Action (RFA) conducted field work in the spring of 2013 to explore the implementation 
and impact of the VPT on developmental education instruction and completion. Through interviews 
with math and English developmental education faculty and administrators, and focus groups with 
current developmental education students, researchers were able to gather data on the successes and 
challenges faced by the college as it implements the VPT and newly designed developmental courses 
(see Table 3). 


9 Virginia Community Colleges Annual Report: 2011-12 

“Achieving the Dream (2013). Enrolling in a developmental math summer bridge program at danville community college. Retrieved from: 
http://www.achievingthedream.org/resource/enrolling in a developmental math summer bridge program at danville community col 
lege 

11 Achieving the Dream (2012). 
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Table 3: Research Conducted 


DATA SOURCES 

Interviews with Administrators 3 

Interviews with Developmental Education Math Faculty 3 

Interviews with Developmental Education English Faculty 3 

Developmental Education Students Focus Groups 2 


III. Integration of Placement and Diagnostic Assessment at the Community College 

Danville Community College administers only the Virginia Placement Test (VPT) in math and English 
for both placement and diagnostic purposes, as required by the Virginia Community College System 
(see Table 4). Further findings regarding integration of the VPT are outlined below: 


Key for Table 4 through Table 6 

©YES 

( 

s 

) NO 


Table 4: Level of Diagnostic Assessment Integration within the Community College 


STATE-DEVELOPED ASSESSMENT (VPT) 

Administered in Math? 

O 

Administered in English? 

O 

Fully Implemented with All Students? 

O 


Integration across Subjects and Students: The VPT has been fully implemented across the college 
and informs curriculum in both math and English developmental education courses. The assessment is 
used with all students as they enter the college. In math, the results determine which of the nine math 
modules students will be required to complete, if any. In English, the VPT places students into one of 
three levels of developmental English: ENF 1, ENF 2, or ENF 3 (in which students are also enrolled in 
the introductory, credit-bearing English course). 


Impact on College Policy and Practice: The VPT and aligned curricula have considerably altered the 
practices of DCC. The developmental education curricula have been changed to nine online modules in 
math without the use of a traditional lecture format, and three courses in English with reading and 
writing skills combined into one course. The VPT also has a “floor,” or score beneath which a student is 
considered in need of Adult Basic Education and is ineligible for developmental education coursework. 
Further, it eliminated what had been a common practice with the COMPASS placement test of allowing 
students close to the upper cut score to be placed in the level above; this is no longer the case as the 
scores themselves are not available. One faculty member explained that often an advisor might say, 
“Oh, you were kind of close, so we’re going to go ahead and place you into the next level up;” faculty 
interviewed agreed that this was a common practice. 
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Impact on Instruction Due to the limited data available to faculty, the diagnostic aspect of the test has 
not influenced instruction. Faculty members are not able to use the data to inform their pedagogical 
strategies or differentiate their instruction for particular students. 

IV. Assessment Data Use 

The VPT was designed as both a placement and diagnostic assessment system to determine whether 
students are eligible for credit-bearing courses and, if not, at what level of developmental education 
they should be placed. Our findings regarding the use of the assessment data follow (see Table 5): 

Table 5: Use of Assessment Data 


STATE-DEVELOPED ASSESSMENT (VPT) 

MATH 

ENGLISH 

Are the test results included in the college student data system? 

O 

O 

Is the test providing data that can be used to identify individual student 
learning needs? 

O 

® 

Are the test data linked to other student records? 

® 

® 

Are the data being used outside of student placement and diagnostic 
decisions in developmental education? 

® 

® 


Inclusion in the Student Data System: Placement results are automatically uploaded to the college’s 
student record system, PeopleSoft. After a student takes the math or English VPT, typically within 12 to 
24 hours, those scores are in the system. Students then meet with a counselor who explains the scores 
and what they will mean for the student, and what developmental education courses are needed. 

Identification of Individual Student Learning Needs: Faculty and students do not receive diagnostic 
information from the VPT on the subtopics covered in the math modules or English courses in which 
students have been placed (see Table 5). While the test in math provides more specific information on 
areas of student need by placing them in content-based modules instead of larger math courses, the 
VPT report only lists whether or not a student has been “exempted” from a module, but does not 
provide a test score for each module or outline the skills in the modules that students do or do not 
know. One math faculty member stated that the [VPT results are] “not fine-grained at all... [students] 
either exempted a module, or they didn’t, and if they exempted a module, it’s not broken down to 
individual objectives under that module.” The results in English are even vaguer, reporting only the 
English course where the student is being placed. 

Linkage to Other Student Data: The VPT is not integrated with other student data, so analyses to 
determine the connections between placement scores and other student indicators not captured on the 
VPT are difficult. 

Data Use outside of Developmental Education The VPT data is not being used for purposes outside 
of placement presently. While administrators recognized the need for further analysis, the VPT results 
are not being used by the college for larger analytic or evaluative purposes (see Table 5). One 
administrator explained that the college is “just trying to get it to work for placement. If we can get it to 
work well for placement, then we can look at other ways we can use the information that’s available.” 
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V. Effectiveness of the Assessment System and Developmental Education Program 

The effectiveness of both the assessment system and the reforms in developmental education 
instruction as compared to the systems in place prior to these initiatives provide the basis for evaluating 
success (see Table 6). Our discussion of the findings on that score follows: 


Table 6: Respondent Perceptions of the Effectiveness of Assessments and Curriculum Redesign 


STATE-DEVELOPED ASSESSMENT (VPT) 

MATH 

ENGLISH 

Is the test placing students more accurately in the developmental education 
sequence? 

O 

Mixed 

Is the curriculum redesign allowing students to complete the developmental 
education sequence more quickly? 

O 

Unknown 


Student Placement 

- Math: There has been improvement in the placement of students in developmental math, as 
evidenced by the pre-tests faculty give to students once enrolled in their modules to verily the 
VPT placement results. Faculty found that the VPT is “placing [students] well.” All three math 
faculty members interviewed reported that students have been properly placed in the correct 
modules based on the pre-test given to students at the beginning of the module. One faculty 
member explained that: 

We offer the same test that we give them at the end of the module at the very beginning of a 
module, the student takes that test, just to verify that in fact they do not know that 
material, or they need help with that material and it’s very rare that we have a student 
pass that test... it’s a good check. 

Further, all of the students involved in the developmental math focus group felt that they were 
accurately placed in developmental math and were not surprised by the modules they were 
required to complete. 

- English: While interview and focus group respondents in English felt that the new assessment is 
somewhat improved over the COMPASS, it is still early in the process as the VPT in English was 
first administered at DCC in the fall of 2012. One English professor explained that the VPT is 
aligned with the student learning outcomes and therefore with what they are being taught in the 
classroom in preparation for college-level English. The effectiveness of the test in placing 
students, however, is not confirmed by a pre-test in English as in math; the department is in the 
process of developing such a test. Flowever, all of the students in the developmental English 
focus group felt that they had been properly placed and that the developmental English courses 
they were taking had value for them. 

Developmental Course Completion 

- Math: Students can move through the math modules more quickly and successfully than they did 
when the developmental education curriculum required an entire semester to complete because 
they can exempt out of the topics in which they are already proficient. There was consensus 
among faculty members that the process of completing the developmental education 
requirements is “much faster than it was” and in some cases “students just blaze through these 
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modules.” A faculty member explained that: 


What I think is great for the students is that they are allowed to exempt out of material that 
they already know. ..in our old system , they may have had two semesters of developmental 
math. Now , they can do one unit of math in four or eight weeks... instead of having to spend 
a year to a year and a half in developmental math, they can be done in a semester and just 
get those core topics. 

Further, faculty members reported that students who complete the modules tend to successfully 
pass the required test at the end of the module as well, with one reporting that “students who 
finish their unit typically pass their test... it’s very rare for me to have a student not. They 
eventually finish that unit and typically test right out of it... they typically complete.” 

- English: It is too early in the reform in English to report on whether students move through 
developmental English more quickly. 

Supports to Increase Effectiveness 

- Collaboration with similar community colleges was found to be the most valuable type of 
support to faculty and administrators in implementing new curricula. During convenings in 
support of the initiative, once the formal program ended, the big takeaways for college faculty 
and administrators came in conversations with colleagues about how the test and modules were 
being implemented on other campuses. One administrator explained that “the highlight of the 
meeting for us was interacting with everyone else to see ‘how are you managing to teach three 
units in one classroom?’ Those takeaways are good for us.” One faculty member said that “all the 
developmental math people [get] together and it typically turns into, ‘Okay, what are you doing? 
How is your college handling this?”’ Another remarked that “faculty... would actually 

gather.. .and hash [things] out. ..the state did a great job of getting us together... that was great.” 

- Bedford/St. Martins, a publisher of educational resources, worked with the college to customize 
their online curricular tools and materials to align with the state’s student learning objectives 
(SLOs) in developmental English. Developmental English courses have integrated reading and 
writing, but needed to develop curricula. One member of the English faculty explained the 
process the college went through with Bedford/St. Martin to develop appropriate instructional 
resources: 

Bedford/St. Martins allowed us to put together a custom text that we could use with our 
students that used reading and writing, because we didn’t have a good combined text yet 
available... they were willing to take the computer program they have. ..and our SLOs for 
the VCCS English curriculum and tailor the program so that we have activities in their 
Writing Class program that fits with each of our SLOs, so that we know that we’re 
covering them all. 

VI. Challenges to Implementation 

With the implementation of any new reform, challenges and unintended consequences exist. Faculty 
members and administrators identified a number of barriers in using the assessment. Our findings 
related to these challenges follow: 
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Faculty members are not provided with diagnostic VPT data on the areas of proficiency within 
each of the nine math modules and three English courses. As mentioned previously, the scores 
from the VPT simply list whether students are “exempt” from a math module or English course, but do 
not provide any further data on how well the students performed on the learning outcomes included in 
each. Faculty are therefore not able to use the data to help them focus on the needs of individual 
students, but are simply informed as to which of the modules or courses students will be required to 
pass in order to enroll in credit-bearing courses. One developmental math faculty member explained 
that “it would be very interesting to see.. .like fractions, maybe they knew how to do everything except 
add and subtract fractions... that particular piece of information would be helpful. I would know how to 
work with them when they got there.” 

Without training on the new form of instruction, math faculty struggled with shifting 
instructional styles from lecture to online module facilitation with students working on multiple 
modules in the same classroom. Using an “emporium style” in which students from many module 
levels are in the same classroom, as opposed to more traditional lectures to students at the same level, 
has been a challenging transition for faculty members. One faculty member explained that, “when I 
have students who are doing modules l, 2 and 3 [in the same classroom]... it’s very difficult.” Another 
faculty member said: 

The way we decided to approach it was the emporium style, where everyone’s in the same 
classroom working on different units, different sections, at the same time and basically I’m the 
facilitator. I go around and individually tutor and help them. ..and I’ll be honest, I hated it. ..I 
wanted to make sure that I was showing them what they needed and [felt like I was] not a 
teacher anymore. 

This differentiation of instruction requires that faculty “adapt on the fly” and “really get a feel for your 
students, and... connect with where they are.” As a result of having students from multiple modules in 
the same classroom, “it’s not uncommon for [faculty] to go from teaching factoring to 
teaching. ..decimals.” Despite this shift, the faculty did not receive training in this new form of 
instruction. According to one faculty member, “we really didn’t receive any particular kind of 
training. ..it was almost like they said, ‘here are your new modules. We’re not going to tell you how you 
have to teach them.’” 

Students and faculty often have difficulty covering all of the content included in the math 
modules and English courses in the timeframe recommended by the system office. Math faculty 
and campus administrators reported that while successfully completing and testing out of a module is 
common and takes place more quickly than with the traditional semester courses, it often takes six to 
eight weeks for students to do so. One faculty member explained that “it might take them 6 weeks, it 
might take them 8 weeks. On a very rare case, it might take them 12 weeks to finish a unit.” An 
administrator reported that the “math faculty say students need more than four weeks to get through 
some of the units, particularly the higher units.” Similarly in English, the combination of reading and 
writing content in the same course has made covering the content a challenge over the semester. One 
English faculty member explained that “we are feeling lots of pressure. This has been difficult to do; we 
have folks who have always been full-time reading faculty, others who had writing specialties, so we’ve 
had to help people combine those two things.” In describing the integration of reading and writing in 
the same course, another English faculty member said that “a challenge is an understatement... forcing 
that much information onto a developmental education student.” 
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The community college system office has provided limited direct support to the college on the 
implementation and use of the VPT, instead working through campus liaisons that were 
responsible for campus-level training. While the state has provided opportunities for professional 
development through webinars and developmental education symposia, administrators and faculty 
reported that the system office primarily provided guidance on “new hard and fast rule[s]” as opposed 
to training in support of the initiative. Campus liaisons who were more closely involved with the 
development of the curriculum and test served as the primary line of communication from the state to 
the college, but were given few resources to use in training faculty members. “The folks that we have as 
our liaisons with the VCCS come back to campus and take the lead,” said one administrator. Another 
administrator stated that, “I think we could have gotten more training from the VCCS, because we were 
training about something that we didn’t know that much about at that point.” 

The role of the community college as an open-access institution has been challenged by the 
“floor” created with a cut score below which students are not eligible for developmental 
education courses and cannot receive financial aid. Until the implementation of the VPT, all high 
school graduates or students with GEDs were eligible to enroll at DCC. Now, students scoring below the 
developmental education cut scores on the VPT must take adult basic education classes before entering 
the college as students. To address this issue, faculty at DCC sought out funding to create short “bridge 
courses” between semesters to provide extra assistance to prospective students scoring at the adult 
basic level on the VPT, and have found success in preparing students to score well enough on the VPT 
the second time to become eligible for developmental education courses. 


Conclusions 

The site visit to DCC allowed researchers to look at the implementation of the VPT through the lens of a 
mid-sized campus with students looking primarily for workplace skills, as opposed to those seeking to 
transfer to four-year institutions. The college has implemented the assessment and aligned curricular 
reforms across both subjects and all classes. While the assessment and curricular reforms in math seem 
to be bearing fruit in terms of improvements in placement accuracy and completion, the story in 
English is still developing and more questions remain. Further, instructional challenges for faculty and 
students in content coverage along with concerns around student access since the creation of the “floor” 
have meant that the college has needed to adapt to the new policies and practices they face. Additional 
support from the system office moving forward may be helpful as challenges continue in the 
implementation of the reform. 
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RESEARCH for ACTION 


Developmental Education and Diagnostic Assessment Reform 
Community College Case Study: Northern Virginia Community College (NOVA) 

The following profile has been developed to outline the 
local context and implementation of developmental 
education and diagnostic assessment reform at Northern 
Virginia Community College. The profile was verified by 
the point of contact at the college; data included in the 
profile offers a stand-alone summary and will be used for 
cross-site analyses. The profile addresses six areas: l) local 
and community college context, 2) overview of the 
research, 3) integration of placement and diagnostic 
assessment, 4) diagnostic assessment data use, 5) 
perceptions of the effectiveness of the assessment system 
and developmental education program, and 6) challenges 
to implementation. The following research questions 
guided the site visits at the community college: 

• How do external calls for assessment and curricular 
alignment (e.gf., state postsecondary readiness 
standards) affect institutional offerings? 

• What is the role of additional training and support— 
from states, coordinating boards, K-12, workforce 
partners, and other stakeholders— provided to 
institutions? 

• How tightly coupled are institutional programs and 
diagnostic assessments? In what ways do institutions 
incorporate diagnostic assessment data in student-level 
data systems? 

• Do implementation challenges differ when state- 
developed diagnostic tests are used versus those 
purchased from national vendors? Along what 
dimensions, and why? 

• What are the perspectives of administration, faculty, 
and students concerning the effectiveness of diagnostic 
assessment systems? What factors do institution 
stakeholders cite as barriers to successful implementation? 


Virginia State Context 

The Developmental Education Task Force 
(DETF) was convened in 2008 to review 
the system’s developmental education 
policies and increase student progress 
through their courses towards graduation; 
the initiative was supported by 
involvement in the Developmental Education 
Initiative. The DETF provided 
recommendations in the report The 
Turning Point: Developmental Education in 
Virginia’s Community Colleges the following 
year. In 2010, the Developmental 
Mathematics Redesign Team (DMRT) 
released recommendations for 
developmental mathematics courses across 
the Virginia Community College System 
(VCCS), proposing that the content of 
developmental mathematics curriculum be 
revised with different pathways contingent 
on a student’s program of study, that 
content be organized into nine pre-college 
units of study, and that the VCCS develop 
new placement and diagnostic instruments. 
The Developmental English Redesign Team 
(DERT) proposed that developmental 
English be restructured as an integrated 
reading and writing system, with three 
direct pathways to credit-bearing English 
determined by a student’s placement test 
score and performance against specific 
learning outcomes. 


52 



I. Local and Community College Context 
Local Context 

Northern Virginia Community College (NOVA) serves the far Northeast region of Virginia that borders 
Washington D.C. Among a population of 2.3 million people, 91.3 percent have graduated high school 
while over 50 percent hold a 4-year degree. The population in the region is primarily white (68.8 
percent) while the other three minority groups range from 12.5 to 16.5 percent of the population. 
Economically, the region has a high average household income of $96,272, which exceeds the state 
average of $63,302 (see Table 1). 

Table I: Aggregate Population & Demographics served at NOVA Campuses 

POPULATION 


Total Population 

Percentage of Population with High School Diploma 
Percentage of Population with Four-Year Degree 
Average Household Income 

2,309,342 

91.3% 

54.8% 

$96,272 

DEMOGRAPHICS 

Percentage White 

68.1% 

Percentage African American 

12.4% 

Percentage Hispanic 

16.5% 

Percentage Asian 

13.9% 


Source: U.S Census Bureau State & County Quick Facts at httD://auickfacts. census.gov/qfd/maDs/virginia map.html 


Community College Context 

Founded in 1964, NOVA has grown to be the largest educational institutional in Virginia and the second 
largest in the United States. 12 The college includes a total of six campuses and four educational centers 
that stretch throughout Northeast Virginia. Within those campuses, students have access to 77 two-year 
degree programs and 76 certificate programs (15 Certificate and 61 Career Studies Certificate). 

Table 2, below, shows NOVA’s enrollment data and the academic programs offered. The 35,601 full- 
time students are less than half the total student body of 76,552 students. Within the population, 47.6 
percent of the student body is white, while 34.6 percent are minority non-African American. 
Additionally, nearly 50% of full-time beginning students receive financial aid. 


12 About NOVA: https : / /www. nvcc.edu / about-nova / index.html 
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Table 2: Student Characteristics and Academic Programs 


STUDENT CHARACTERISTICS 

Unduplicated Headcount 76,552 

Total Full-Time Equivalent (FTE) Enrollment 35,601 

Percentage White Students 47.6% 

Percentage African-American Students 17.8% 

Percentage Other Minority 34.6% 

Full-Time Beginning Undergraduates Receiving Financial Aid 47% 

ACADEMIC PROGRAMS 

Number of Bachelor’s Degree Programs (and Graduates) 0 

Number of Associate Degree Programs (and Graduates) 77 (5452) 

Number of Certificate Programs (and Graduates) 76 (2524) 

Number of Diploma Programs (and Graduates) 0 


Sources: NOVA - Facts about NOVA; Virginia Community Colleges: Institutional Research Information and National Center for 
Education Statistics at http://nees.ed.gov/collegenavigator/?id=232Q46#finaid 

NOVA has several partnerships focusing on workforce development, SySTEMic Solutions, (STEM) and 
NoVaHealthyFORCE (health workers). 13 The college has also partnered with Achieving the Dream 
(ATD) since 2007. Through their work with ATD, NOVA has improved retention rates through new 
student orientation, advising, and registration programs. In addition to improving a “student’s entering 
experience,” ATD has also helped with redesigning the developmental math curriculum with support 
from the National Center for Academic Transformation. 14 

II. Overview of the Research 

Research for Action (RFA) conducted a field work visit at the NOVA Manassas Campus in the spring 
semester of 2013 to explore the implementation and impact of the Virginia Placement Test on 
developmental education instruction and completion. Through a mix of interviews with developmental 
education faculty in math and English, college administrators, and focus groups with current NOVA 
students at the Manassas Campus, we were able to explore the successes and challenges faced by the 
college as it implements the VPT and newly designed developmental courses. 

Table 3 below breaks down the number of interviews and focus groups that informed the analysis 


highlighted throughout this report. 

Table 3: Research Conducted 

DATA SOURCES 

Interviews with Administrators I 

Interviews with Developmental Education Math Faculty 2 

Interviews with Developmental Education English Faculty 4 

Developmental Education Student Focus Groups 2 


'3 NOVA Partnerships: https://www.nvcc.edu/about-nova/partnerships/mdex.htmI 
Achieving the Dream: Northern Virginia Community College 
http://www.achievingthedream.org/college profile /northern Virginia community college 
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III. Integration of Placement and Diagnostic Assessment at the Community College 

NOVA only administers the Virginia Placement Test (VPT) in math and English, as required by the 
Virginia Community College System (see Table 4). At NOVA, the VPT replaced COMPASS. Our findings 
regarding the integration of the reform at NOVA are outlined below: 


Key for Table 4 through Table 6 

©YES 

( 

S 

) NO 


Table 4: Level of Diagnostic Assessment Integration within the Community College 


STATE-DEVELOPED ASSESSMENT (VPT) 

Administered in Math? 

O 

Administered in English? 

O 

Fully Implemented with All Students? 

O 


Integration across Subjects and Students: The VPT is required across the VCCS system and so is 
implemented with all native English-speaking students as they enter the college to place them in both 
math and English. 

Impact on College Policy and Practice: 

- English: Two changes were identified regarding curricular practice in developmental English: 1) the 
reduction of the number of credit hours, especially at ENF2 (four credits) as a result of the 
integration of reading and writing courses and 2) elimination of the placement mechanism for the 
English Honors course. 

1) Many faculty mentioned ENF2 as the biggest challenge because they have to cover reading 
and writing all in a shortened period of time. One English faculty member explained that she 
has less time to help her ENF2 students individually: 

With the ENF 2 , 1 have less time to spend individually hands on with my students. For 
some students , that’s okay , and for others, that personal contact that they have with 
that faculty member is so valuable to motivating them and helping them understand 
the value of what they’re learning and to develop as a student. So that’s been a big 
change. 

The redesign condensed a five to 10 credit class (some students would take five credit reading 
and five credit writing and some would take only one) to a four credit class that combines both 
reading and writing. 

2) Students were placed in the Honors English class based on their COMPASS scores. Since the 

COMPASS test has been replaced by the VPT, there is no placement mechanism for Honors 
English placement. NOVA faculty members are considering placing students who scored 
high in ENG111 in Honors. 
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- Math: Due to the size of the college, NOVA implemented the math modular approach in an MTT4 
format, meaning that four different modules may be covered in one semester. 

Impact on Instruction: English faculty expressed concerns about the lack of diagnostic information on 
VPT and the impact it has on instructional practice. One English faculty member explained that the 
community college system office “sold” McCann and the VPT to the faculty by explaining that it would 
provide great diagnostic information. However, this was not the case: 

The Chancellor , when he announced that they were going with McCann and Associates said 
that one of the great things about this new test was it was going to give us. ..a good way to see 
exactly where the students’ strengths and weaknesses were. In other words, we were promised 
good diagnostics. We did not get good diagnostics. If you’ve looked at the test, if you see what 
is printed out, it’s a general rubric. 

Faculty stated that they want the students’ strengths and weaknesses “broken down as much as 
possible,” and a copy of the essay so they can see how the student writes. 

IV. Assessment Data Use 

The VPT was designed as both a placement and diagnostic assessment system to determine whether 
students are eligible for credit-bearing courses and if not, at what level of developmental education they 
should be placed. Our findings regarding the use of the assessment data follow: 


Table 5: Use of Assessment Data 


STATE-DEVELOPED ASSESSMENT (VPT) 

MATH 

ENGLISH 

Are the test results included in the college student data system? 

O 

O 

Is the test providing data that can be used to identify individual student 
learning needs? 

O 

® 

Are the test data linked to other student records? 

® 

® 

Are the data being used outside of student placement and diagnostic 
decisions in developmental education? 

® 

® 


Inclusion in the Student Data System: Testing information is automatically uploaded to the colleges’ 
student record system, but faculty only see placement results. The reporting system for the VPT in 
English provides students with a printout of their English score and their essay score. The English score 
is presented as a simple score of a student’s multiple choice answers, and the essay score is broken 
down into five components outlining a student’s strengths and weaknesses. However, when it is loaded 
in PeopleSoft, the scores are combined and reported only as placement decisions: “below the floor”, 
“ENFi”, “ENF2”, “ENF3”, and “on-level.” In math, the VPT reports which of the nine skill-based 
modules students are required to complete and the placement directly corresponds to the modules. 

Identification of Individual Student Learning Needs: Faculty stated that if they want to see individual 
students’ reports, they would request that from the testing center as it is not available through their 
student record system. The VPT in English does not provide diagnostic information for the 
identification of student academic needs. Although the VPT in math provides some diagnostic 
information, math professors are hesitant to trust that information as the scoring system is not 
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transparent. Faculty and staff don’t know the score ranges or the cut scores that are used to place 
students into courses. The determination of cut scores, in one professor’s words, “was a huge mystery.” 

Linkage to other Student Data: VPT data is not linked to other student data records and so cannot be 
used for larger analyses with other student information. 

Data use Outside of Developmental Education At this point, the VPT is only used to place students. 
While in the past, placement test results were used for advising students, these data do not lend 
themselves to that process, particularly in English because faculty and counselors don’t receive any 
diagnostic information on the student. 

V. Effectiveness of the Assessment System and Developmental Education Program 

Faculty, student and administrative perspectives on the effectiveness of the VPT assessment system and 
curricular redesign are summarized below. As shown in Table 6, faculty and administrators were mixed 
as to the effectiveness of the VPT and aligned curricular redesign. 


Table 6: Respondent Perceptions of the Effectiveness of Assessments and Curriculum Redesign 


STATE-DEVELOPED ASSESSMENT (VPT) 

MATH 

ENGLISH 

Is the test placing students more accurately in the developmental education 
sequence? 

( 

N 


( 

N 


Is the curriculum redesign allowing students to complete the developmental 
education sequence more quickly? 

© 

Mixed 


Student Placement: Overall, math and English faculty as well as the administrators indicated that “a 
disproportionately high percentage of students are placing into college level courses.” Further, math 
and English faculty members often stated that they’d like to return to the COMPASS, not because it was 
a better test but because they had local control over the placement of students. 

- Math: One math faculty member explained that their teaching practice changed because 
students are misplaced. Particularly for students who place in the upper levels, math faculty 
explained that students appear to not have the necessary math foundation. For example, a 
faculty member explained that in his Pre-Calculus class, he was shocked at how little students 
knew: 

“I went back and looked up every single student to ensure that they did actually take the 
placement test. And they did, either they passed the placement test, or they took a 
placement test and went to MTT and came out of the MTT. And it’s just letting students in 
that can’t do the most basic things.” 

- English: A large number of students are placing into credit-bearing English despite the common 
belief among faculty interviewed that the VPT is a more difficult test than COMPASS: 

“The thing that was really shocking about the test, after we took it we were like, oh my 
goodness, this is difficult, the students aren’t going to be able to pass, we’re going to have 
so many students in developmental English. But then when the students started taking the 
test, everybody was getting into ENG ill after taking the test.” 


57 


Similarly, an English faculty member stated that, “I look at the ENF2 entry-level competencies; I 
have to teach most of those, because the students aren’t bringing that knowledge.” Another 
faculty member added that the VPT has ’’impacted my teaching because you think you’re going 
to come in one way and do something, but in reality you have to bring it down a few notches to 
get to where they need to be. They need the basic, basic, basics.” 

Developmental Course Completion: The redesign reduced the time it takes students to enroll in 
credit bearing courses in both math and English (provided, of course, that the students pass these 
developmental courses in a timely fashion.) The English redesign combined both reading and writing 
courses into three placement levels, ENFi, ENF2, and ENF3. ENFi is offered as eight credits, ENF2 is 
four credits, and ENF3 is two credits but taken in conjunction with the introductory credit-bearing 
English course which is three credits. However, concerns over placement accuracy and the recent 
implementation of the redesign led to mixed responses as to whether students will be able to complete 
developmental English more quickly. The math redesign eliminated core content and pre-requisites, 
and established different exit points. Whereas STEM majors have to pass out of ah nine modules, liberal 
arts majors only need up to module 5 and applied majors only need modules 1 through 3. 

Supports to Increase Effectiveness 

- The VCCS has provided limited support in terms of VPT implementation. The VPT is mandated 
by the VCCS as the placement test for ah Virginia community colleges. Despite this, 
administrators and faculty report that they were not provided with sufficient information to 
read and utilize the results; some claim they were provided with “nothing from the state as far as 
the diagnostic test is concerned.” The testing center is in communication with a state liaison for 
technical issues or other concerns, but there was no faculty mention of direct state contact. 

- The NOVA administration has created a system of supports on each campus. One English and 
one math faculty member is designated on each campus as the Campus Implementation Leads 
(CILs) and acts as a resource for other faculty members. Professional development is planned by 
the CILs to address faculty issues and concerns. Multiple English developmental faculty 
members also reported that they served on the Developmental English Redesign Advisory 
Committee at NOVA. This committee also includes deans, vice presidents, ESL faculty, and 
credit English faculty. 

VI. Challenges to Implementation 

With the implementation of any new reform, challenges and unintended consequences exist. Faculty 
members and administrators identified a number of barriers in using the assessment. Our findings 
related to these issues follow: 

There is widespread dissatisfaction at NOVA with the VPT as compared to the COMPASS 
placement test. Comments from faculty and administrators ranged from “I’d like to see the VPT 
scrapped or significantly changed” to “I think the VPT could be better if it had more work. The 
COMPASS was a better test, but only because we had more control over it, we could make it fit our 
needs better.” Math faculty members felt that the diagnostic element of the VPT is not yet fully realized 
and that the test questions can be adjusted to better diagnose student needs’ and place them in the 
correct modules or courses. 
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The VPT removed any local control over the placement of students due to the lack of test 
scores. An English faculty member expressed this concern by stating that, “I think we had a good 
system in place in looking at multiple measures and looking at other factors, to try to move students 
into credit classes when they were ready.” While the VPT was advertised as a customized assessment, it 
resulted in less customization and flexibility at the local level. Another faculty member stated: 

COMPASS is more effective because it gave me scores; it gave me a range. ..so I would say 

COMPASS helped us to know which areas [students] were weak in, because it gave a score. 

With the VPT it doesn’t tell us anything except [placement] . No more moving. Now, once 

they’re placed they’re stuck. 

The increased length of time to take the practice VPT as well as the test itself has made the 
administration less efficient. Personnel at NOVA stated that the VPT is less efficient than the former 
placement system because it takes longer to administer. As one interviewee stated “It’s also been 
challenging because the math test can be really long... upwards of four to five hours.” Students also 
complained about the length of the VPT online practice materials. “The problem is it gives you ah the 
material from the equivalency of the starting point ah the way up to the highest point you could take. 
You never really knew what you should be studying.” Similarly, an English faculty member related that 
a long process is involved in gaining access to the online practice materials and ended by saying “if I 
were a student I don’t know if I would go through ah this rigmarole to do that.” 

The quality of the VPT itself was sharply criticized. Most of the interview respondents took parts of 
the VPT and cited a number of problems, including typos, poorly written items, ambiguous answers, 
formatting problems, archaic language and outdated reading passages. In addition, system glitches 
were identified. An administrator explained that “our testing centers are constantly adjusting the 
instructions they give to students so they can add things like, don’t hit the tab key because the system 
won’t read it and thinks you’re writing in a foreign language and won’t score your essay.” 

Conclusions 

The individuals interviewed at NOVA were sharply critical of the VPT in both English and math. The 
English faculty members were particularly troubled by the lack of diagnostic information in the 
assessment and by the curriculum redesign. Math faculty members were less concerned by the new 
curricula but shared their colleagues’ misgivings about the VPT. However, ah had serious reservations 
about the changes, believing that the new system is less effective and efficient than the former system. 
Although most acknowledge that the changes are still very new and thus can’t be completely evaluated, 
they are acutely aware of the problems and concerned that students are being harmed. One faculty 
member said “I wake up [at night] thinking about these students, and what’s going to happen to them, 
and the fact that so many of them are not doing well, and they’re not going to be ready to move on.” 
Whether the new system helps or hinders students and at what scale is a question that remains to be 
answered. 
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RESEARCH for ACTION 


Developmental Education and Diagnostic Assessment Reform 
Community College Case Study: Patrick Henry Community College (PHCC) 

The following profile has been developed to outline the 
local context and implementation of developmental 
education and diagnostic assessment reform at Patrick 
Henry Community College. The profile was verified by the 
point of contact at the college; data included in the profile 
offer a stand-alone summary and will be used for cross-site 
analyses. The profile addresses six areas: l) local and 
community college context, 2) overview of the research, 3) 
integration of placement and diagnostic assessment, 4) 
diagnostic assessment data use, 5) perceptions of the 
effectiveness of the assessment system and developmental 
education program, and 6) challenges to implementation. 

The following research questions guided the site visits at 
the community college: 

• How do external calls for assessment and curricular 
alignment (e.gf., state postsecondary readiness 
standards) affect institutional offerings? 

• What is the role of additional training and support— 
from states, coordinating boards, K-12, workforce 
partners, and other stakeholders— provided to 
institutions? 

• How tightly coupled are institutional programs and 
diagnostic assessments? In what ways do institutions 
incorporate diagnostic assessment data in student-level 
data systems? 

• Do implementation challenges differ when state- 
developed diagnostic tests are used versus those 
purchased from national vendors? Along what 
dimensions, and why? 

• What are the perspectives of administration, faculty, 
and students concerning the effectiveness of diagnostic 
assessment systems? What factors do institution 
stakeholders cite as barriers to successful implementation? 


Virginia State Context 

The Developmental Education Task Force 
(DETF) was convened in 2008 to review 
the system’s developmental education 
policies and increase student progress 
through their courses towards graduation; 
the initiative was supported by 
involvement in the Developmental Education 
Initiative. The DETF provided 
recommendations in the report The 
Turning Point: Developmental Education in 
Virginia’s Community Colleges the following 
year. In 2010, the Developmental 
Mathematics Redesign Team (DMRT) 
released recommendations for 
developmental mathematics courses across 
the Virginia Community College System 
(VCCS), proposing that the content of the 
developmental mathematics curriculum be 
revised, with different pathways contingent 
on a student’s program of study; that 
content be organized into nine pre-college 
units of study; and that the VCCS develop 
new placement and diagnostic instruments. 
The Developmental English Redesign Team 
(DERT) proposed that developmental 
English be restructured as an integrated 
reading and writing system, with three 
direct pathways to credit-bearing English 
determined by a student’s placement test 
score and performance against specific 
learning outcomes. 
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I. Local and Community College Context 
Local Context 


Patrick Henry is located in Martinsville, Virginia, and serves students from Martinsville, Henry County, 
Patrick County, and the southern portion of Franklin County. Martinsville once used to be considered 
the sweatshirt capital of the world, with a large agricultural middle class. The city’s main industries for 
many years were large textiles and the furniture industry. Beginning in the mid-to-late 1990s, however, 
the textile and furniture industries were vacated as manufacturing became economically unsustainable. 
Currently, Martinsville has approximately a 15% unemployment rate, 15 with about 24% of the 
population below poverty level. 16 The main employment drivers of the community are medical, service, 
manufacturing, and social service sectors. Table 1 provides an aggregate overview of the economic 
conditions and the population of Martinsville, Henry County, Patrick County, and Franklin County. 

Table I: Average Population & Demographics for the City of Martinsville, Henry County, Patrick County, and 
Franklin County 


POPULATION 

Total Population 

141,564 

Percentage of Population with High School Diploma 

77% 

Percentage of Population with Four-Year Degree 

15% 

Average Household Income 

$37,450 

DEMOGRAPHICS 

Percentage White 

77% 

Percentage African American 

20% 

Percentage Hispanic 

3% 

Percentage Asian 

0.6% 


Source: U.S Census Bureau State & County Quick Facts at http://qu ickfacts.census.gov/ 


Community College Context 

Patrick Henry Community College is a small two-year institution that serves approximately 3,000 
students. It offers 29 associate degree programs, 12 certificate programs, and 38 career studies 
certificate programs. Serving a variety of community members, Patrick Henry also offers a variety of 
nontraditional programs such as customized workforce development training and industry-recognized 
certifications. The college has a long-standing history of developmental reform, as a high percentage of 
their students go through developmental education. 

Patrick Henry Community College was founded in 1962 as a branch of the University of Virginia’s 
School of General Studies, and became part of the Virginia Community College System in 1971. One of 
Patrick Henry’s stated goals includes identifying, meeting, and integrating the needs of the community 
it serves, as well as meeting the workforce needs of the community. In addition to serving students 
aspiring to transfer to a four-year institution, Patrick Henry serves those who may be seeking trade- 
specific certificates or associate degrees. Therefore, they offer “an extensive curriculum of credit and 
non-credit technology, business, and professional development programs.” 1 ? 

Table 2: Student Characteristics and Academic Programs 


http://www.bls. gov/roq/valaus.htm 
16 http: //quickfacts. census. gov/qfd/states/ c ;i/ l: ;i6qo.html 
'/ http://www.ph.vccs.edu/ahout 


61 




STUDENT CHARACTERISTICS 

Unduplicated Headcount 

4,714 

Total Full-Time Equivalent (FTE) Enrollment 

2,512 

Percentage White Students 

70% 

Percentage African-American Students 

25% 

Percentage Hispanic Students 

2% 

Percentage Other Minority 

3% 

Full-Time Beginning Undergraduates Receiving Financial Aid 

87% 

ACADEMIC PROGRAMS 


Number of Bachelor’s Degree Programs (and Graduates) 0 

Number of Associate Degree Programs (and Graduates) 29 (457) 

Number of Certificate Programs (and Graduates) 50 (33) 

Number of Diploma Programs (and Graduates) 0 

Source: VCCS Institutional Research: http://mvfuture.vccs.edu/Research/index.html 


Patrick Henry has a history in community college education reform, particularly with developmental 
education. Since 2004, Patrick Henry has been part of the Achieving the Dream Network (ATD), a 
national reform network dedicated to community college student success and completion. Through the 
ATD grant, the college piloted Cooperative Learning, which focuses on student engagement and critical 
thinking, and found successes in the form of improved graduation rates. 18 In 2009, Patrick Henry, 
along with 26 community colleges in the nation, was chosen to receive the Developmental Education 
Initiative grant, which it has used to implement two different strategies: Fast Track math, which allows 
students to progress through developmental math courses more quickly; and the Accelerated Learning 
Program, 19 which allows students who place in the upper level developmental writing course to 
concurrently register for ENG 111, the on-level English course. 20 Accelerating students’ progress 
through developmental education has been the focus of the college since then, and it has in many ways 
positioned the college well for the math and English developmental education curriculum redesign. 
Finally, in 2012, Patrick Henry was one of four community colleges nationwide chosen to participate in 
ATD’s Catalyst Fund grant program, to scale their cooperative learning strategies across the institution 
and to assist other ATD institutions in their efforts at promoting cooperative learning. 21 


18 http://www.martinsvillebulletin.com/article.cfm?ID=3558i 
http:/ /alp-deved.org/about-alp/what-is-alp-exactly/ 

20 http://deionline.blogspot.com/2012/05/patrick-henry-community-college.html 

21 http:/ /www.martinsvillebulletin.com/article.cfm?ID=3558i 
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II. Overview of the Research 


Research for Action (RFA) conducted a field work visit in the spring semester of 2013 to explore the 
implementation and impact of the Virginia Placement Test (VPT) on developmental education 
instruction and completion. Through a mix of interviews with math and English developmental 
education faculty and college administrators, and focus groups with current Patrick Henry students, our 
researchers were able to gain an insight into the successes and challenges faced by the college as they 
implement the VPT and newly designed remedial courses. 

Table 3, below, lists the number of interviews and focus groups that informed the analyses throughout 
this report. 


Table 3: Research Conducted 

DATA SOURCES 

Interviews with Administrators 3 

Interviews with Developmental Education Math Faculty 3 

Interviews with Developmental Education English Faculty 2 

Developmental Education Student Focus Groups 3 


III. Integration of Placement and Diagnostic Assessment at the Community College 


Patrick Henry Community College (PHCC) administers only the VPT in math and English for both 
placement and diagnostic purposes, as required by the Virginia Community College System (see Table 
4). Our findings regarding the integration of the reform at Patrick Henry follow: 


Key for Table 4 through Table 6 





NO 


Table 4: Level of Diagnostic Assessment Integration within the Community College 


STATE-DEVELOPED ASSESSMENT (VPT) 

Administered in Math? 

O 

Administered in English? 

O 

Fully Implemented with All Students? 

O 


Integration across Subjects and Students: The VPT is used across both English and math and 
administered to all students upon entry to the college. 

Impact on College Policy and Practice: The VPT and curricular redesign have had considerable 
impact on the developmental education practices at Patrick Henry. Integrating both reading and 
writing is a new approach to teaching developmental English. With the redesign, English faculty spoke 
of being more involved with all aspects of developmental English than they were prior to the reforms, 
when one faculty member would teach writing and another would teach reading. One faculty member 
explained that s/he has “to acknowledge the link, which is good, because the link between reading and 
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writing is so strong that I think it would be better served to teach both, interlocking them.” Moreover, to 
ensure that each student is ready to progress to the next level, Patrick Henry uses a portfolio system to 
chart developmental students’ progress so that instructors can see previous student work. In math, 
faculty reported less flexibility in determining content and pedagogical strategies due to the 
modularization of math instruction through online tools. 

Impact on Instruction: Without the test score data, faculty members are not able to differentiate 
instruction based on the placement of students in developmental education modules or courses. To 
compensate for the lack of diagnostic data, faculty in ENF l and 2 “give a diagnostic test the first week 
of class to understand students’ strengths and weaknesses.” The ENF3 instructor was also considering a 
move to a diagnostic tool: “I previously thought that there was no need for a diagnostic for ENF3 
because they’re already in English 111. I’m definitely rethinking that next semester because I still need 
to know where their strengths and weaknesses are, even though they’re supposed to be almost college- 
ready.” 


IV. Assessment Data Use 

The VPT was designed as both a placement and diagnostic assessment system to determine whether 
students are eligible for credit-bearing courses and, if not, at what level of developmental education 
they should be placed. Our findings regarding the use of the assessment data follow: 

Table 5: Use of Assessment Data 


STATE-DEVELOPED ASSESSMENT (VPT) 

MATH 

ENGLISH 

Are the test results included in the college student data system? 

O 

O 

Is the test providing data that can be used to identify individual student 
learning needs? 

O 

® 

Are the test data linked to other student records? 

® 

® 

Are the data being used outside of student placement and diagnostic 
decisions in developmental education? 

® 

® 


Inclusion in the Student Data System: Testing information is automatically uploaded to the college’s 
student record system, PeopleSoft, but English faculty only see placement results of “below the floor,” 
“ENFi,” “ENF2,” “ENF3,” and “on-level” in English. Students receive a more detailed report of their test 
results, which include a printout of their multiple-choice English score and a general breakdown of 
their essay score based on five components outlining strengths and weaknesses. In math, the VPT 
reports which modules students are required to complete and the placement directly corresponds to the 
modules. 

Identification of Individual Student Learning Needs: The VPT in math provides diagnostic 
information on student learning needs and is aligned with the math curriculum. This yields a “laser-like 
focus on students’ needs.” The VPT in English does not yield usable or accessible information related to 
the diagnostic identification of student learning needs. English faculty members use Pearson’s My Skills 
Lab Plus to gather diagnostic information that isn’t provided from the VPT. My Skills Lab Plus is used 
to group students together in such a way that they can help each other and reveal students’ strengths 
and weaknesses. 
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Linkage to Other Student Data: VPT data is not linked to other student data records, and so cannot 
be used for larger analyses involving other student information. 

Data Use outside of Developmental Education At this point, the VPT is only used to place students. 
While in the past, placement test results were used for advising students, these data do not lend 
themselves to that process, particularly in English because faculty and counselors don’t receive any 
diagnostic information on students. 

V. Effectiveness of the Assessment System and Developmental Education Program 

The effectiveness of both the assessment system and the reforms in developmental education 
instruction, as compared to the systems in place prior to these initiatives, provide the basis for 
evaluating success (see Table 6). Our discussion of the findings on that score follows: 


Table 6: Respondent Perceptions of the Effectiveness of Assessments and Curriculum Redesign 


STATE-DEVELOPED ASSESSMENT (VPT) 

MATH 

ENGLISH 

Is the test placing students more accurately in the developmental education 
sequence? 

O 

Mixed 

Is the curriculum redesign allowing students to complete the developmental 
education sequence more quickly? 

Mixed 

Unknown 


Student Placement 

o Math: The VPT in math is seen generally as a more-accurate placement assessment than the 
previous COMPASS test, in that it places students into one of nine modules based on content 
knowledge. Faculty members agreed that “the assessment [VPT] more effectively measures 
students’ math levels... because it aligns with the curriculum.” However, there remain concerns 
about why students often test out of modules at higher levels than those they fail; one member 
of the math faculty explained that module 5 is the most difficult module, and yet it is one of the 
easiest on the VPT, so many students can easily test out of it while failing module 4. 

- English: Due to the considerable drop in developmental education student enrollment as a result 
of VPT, several instructors were skeptical of the placement. Though the ENF 1 and ENF2 
English instructors felt that students were accurately placed, this was not the case with ENF3. 
One English faculty member explained that she has students who should have been placed 
lower. Another said that students need to be provided with more support than they were in the 
past: 


I think it really has changed the types of students I got. ..I mean, it shouldn’t, because it’s 
still the same class that it used to be, the material that we cover, what we do and 
every thing... but I am having to do more. ‘You need to study harder, you need to go to 
the writing center, you need to bring a pencil.. .get out your notebook, we’re going to 
take notes now.’ 


65 


Developmental Course Completion: 

o Math: The curricular redesign with the implementation of modularization reduced the time it 
might take students to enroll in credit-bearing courses, but also has resulted in declining passing 
rates. While STEM majors have to pass out of all 9 modules, liberal arts majors only need to 
complete up to module 5, and applied majors only need modules 1-3. Students may select 
whether to enroll in the self-paced “emporium” environment, in which students in multiple 
modules are in the same classes and work only through the online modules, or in a more 
traditional class-based environment. Offering both types is unique to Patrick Henry, as some 
other community colleges offer only “emporium” style classes. 

However, math faculty stated that their pass rates declined after the redesign as well, so the 
developmental math program has shifted practices to compensate in hopes of improving the 
pass rates. One math faculty member stated: 

“We were in good shape before the redesign. We would have sometimes 75-80 percent pass 
rates in some of our classes, especially what we called fast-track classes, where we would 
do algebra 1 in 8 weeks, and then algebra 2 in 8 weeks. We were always running over 70% 
pass rates. And then we were suddenly getting 50-60% pass rates with the new redesign. 
And so we’re like, don’t mess with us, we were okay! But we think we’re finding reasons 
why we’re trying to do some different things.” 

- English: The integration of reading and writing reduced the time it takes for students to reach 
credit -bearing English courses, but there is not enough data to determine if students will be as 
well-prepared for credit-level courses as they were before the redesign. Only time will tell 
whether the VPT and the redesign will yield better success rates in developmental and credit- 
level English classes. As one instructor put it “we won’t know a whole lot until we’re down the 
road and we’re seeing how these students do once they get out of developmental classes.” 

Supports to Increase Effectiveness: Several faculty members discussed the importance of meeting 
their peers across the state and discussing developmental education methods and strategies in general, 
along with the new tests and curricula in particular. In addition, the faculty and administrators at PHCC 
described excellent professional-personal relationships as a crucial support factor during the creation 
and implementation of the VPT and the redesign. 

VI. Challenges to Implementation 

With the implementation of any new reform, challenges and unintended consequences exist. Faculty 
members and administrators identified a number of barriers to using the assessment. Our findings 
related to these challenges follow: 

There was consensus across all interview respondents that the length of the tests is a problem. 
The math and the English tests are taken in two separate sessions and require a minimum of 2 hours 
per session. Students in the focus group felt that the math and the English tests were both very long and 
complicated. Faculty and administrators worried that having separate sessions for the two subject areas 
would prove to be a barrier for students. However, as one interviewee said, “It’s not more efficient, but 
it may be more effective in understanding their needs and placing them in the right place.” 
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PHCC staff voiced challenges in advising students due to the inability to see student scores. 

An administrator explained that her inability to understand how close a student tests related to the cut 
score prevents her from advising students appropriately. Similarly, training faculty advisors on what to 
look at to determine students’ needs based on the VPT scores has been difficult. An administrator 
explained that with the COMPASS test, the College received a certain score which allowed advisors to 
see how close the student was to the cut score, and could work with the student on figuring out what 
resources s/he needed to progress into college-level courses. 

Patrick Henry discontinued established developmental education tools as a result of the 
curricular redesign. A case-management database PHCC used to determine whether a student was a 
good candidate for fast-track developmental math education or regular courses is no longer used on 
campus. It asked a series of questions such as whether the student took math labs, how many times they 
accessed tutoring, and what their COMPASS assessment score was, to assess whether the student would 
be successful in fast-track math courses. Counselors and faculty used it as an advising tool to refer 
students to certain courses as well as resources. However, due to the redesign, this tool is no longer 
used. 

The VPT and the redesign have impacted how the college provides for students placed “below 
the floor.’’ For students who can’t place into math module l or ENFi, they are not able to use their 
financial aid to take courses adult education courses. It was reported that approximately 75 to 85 
percent of the students are on Pell Grants at Patrick Henry, so this policy affects many students’ access 
to the institution. As a result, Patrick Henry decided to combine adult basic education courses with 
other coursework (e.g., include developmental education math skills in a computer class) so that they 
can receive financial aid. PHCC created a 2-credit course for math called the ITE95, which teaches 
students basic computer instruction and whole numbers; and a 2-credit course for English called the 
English 95, which prepares students for ENFi. This way, students can receive their financial aid money 
and take courses at Patrick Henry as opposed to having to take Adult Basic Education courses at the 
Juvenile Detention Center. When describing this policy, an administrator stated: 

We are no longer an open-door institution; the VCCS is no longer an open-door system, 
because they created a floor. We have changed the mission; modified what a community 
college is. We have always said we’ll take you regardless. Now we are, locally, still taking 
students regardless, because we got creative, but there are colleges in the state saying we can’t 
serve you; you’re below the floor. . . there is a strong belief that if you’re way down [below] 
this floor, are you really going to get them out the door? Especially now that with Pell Grant 
for 12 semesters... are you ever really going to get them through? My thinking is if we don’t get 
them out the door, who will? 

Conclusions 

The VPT in Math and English and the respective curricular redesigns carry benefits and constraints for 
students, faculty members, and administrators at PHCC. The benefit of a uniform state-wide placement 
test and course structure is countered by the constraints placed on PHCC faculty and administrators’ 
ability to customize placement for individual students. The benefit of more accurate placement in math 
is counterbalanced by the reduced efficiency of the testing sessions and lower pass rates since the 
redesign. Whether the benefits of the VPT and redesign will outweigh the constraints at PHCC, a college 
that had excellent success rates before these changes, is a question that will be determined over time 
through the examination of student outcomes in credit -level math and English. 
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Appendix B. Local and Community College Characteristics 

Local Context 

Less than 15 percent of the individuals occupying the areas served by Danville Community College (14.3 
percent) and Patrick Henry Community College (14.4 percent) have a four -year degree, which is the 
lowest out of the six colleges in our study. Northern Virginia has the highest percent of four -year degree 
attainment (55 percent). All three of the study colleges in Florida have rates greater than 25 percent 
with Tallahassee having the highest rate (46.5 percent). See Figure lB. 

Figure I B: Percent of Population with a High School Diploma for Four-Year Degree 
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The area around Northern Virginia, which borders Washington D.C., boasts the highest average 
household income. Of the five remaining schools, the average income ranges from $35,971 (Danville 
Community College) to $45,89i(St. Petersburg College). See Figure 2B. 

Figure 2B: Average Household Income 
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School Context 

All six colleges have a mix of certificate and associate degree offerings. Northern Virginia has the 
greatest number of overall programs and associate programs. Valencia College, in Florida, provides 
students with the largest number of certificate program choices. See Figure 3B. 

Figure 3B: Number of Associate and Certificate Granting Programs 
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Following similar trends as the program offering chart, Danville and Patrick Henry community colleges 
have the smallest student bodies, 6,493 and 4,714 students, respectively. Northern Virginia has the 
highest unduplicated headcount of 76,552 followed by Valencia College, 64,912. See Figure 4B. 

Figure 4B: Unduplicated Headcount 
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As for full-time equivalent students, while Northern Virginia has the largest unduplicated headcount, 
St. Petersburg College is comprised mostly of FTE students, 42,342. Around 50 percent of Northern 
Virginia, Valencia College and Patrick Henry Community College’s headcount is attributed to FTE 
students. See Figure 5B. 

Figure 5B: Full-Time Equivalent (FTE) Enrollment 
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For all six colleges Caucasian students comprise the largest percent of the student body ranging from 36 
percent to 70 percent of the student population. African Americans are the second largest student group 
in five colleges while Hispanic students comprise the second highest percent for Valencia College. See 
Figure 6B. 

Figure 6B: Student Body Demographics: Percent of Total Enrollment 
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